Jul 29 21:16:23.383428 ip-10-0-129-123 systemd[1]: Starting Kubernetes Kubelet...
Jul 29 21:16:23.837507 ip-10-0-129-123 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:16:23.837507 ip-10-0-129-123 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 29 21:16:23.837507 ip-10-0-129-123 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:16:23.837507 ip-10-0-129-123 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 29 21:16:23.837507 ip-10-0-129-123 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 29 21:16:23.840165 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.840079    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 29 21:16:23.845011 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.844992    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.845011 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845010    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.845011 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845014    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845017    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845021    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845024    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845027    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845031    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845034    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845036    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845052    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845055    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845068    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845074    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845077    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845080    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845083    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845086    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845088    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845093    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845097    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845100    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.845122 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845103    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845106    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845109    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845111    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845114    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845117    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845119    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845123    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845126    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845129    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845132    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845134    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845137    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845140    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845143    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845147    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845149    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845153    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845155    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845166    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.845615 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845169    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845171    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845174    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845177    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845179    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845182    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845184    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845187    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845190    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845192    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845195    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845198    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845200    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845203    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845208    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845211    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845214    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845216    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845219    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845222    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.846138 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845224    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845227    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845230    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845232    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845235    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845238    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845241    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845244    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845246    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845250    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845252    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845258    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845261    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845264    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845267    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845270    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845273    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845276    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845278    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.846678 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845281    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845284    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845287    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845289    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845292    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845295    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845298    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845303    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845306    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845310    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845313    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845316    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845319    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845321    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845324    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845327    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845329    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.845332    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846132    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.847266 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846141    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846145    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846148    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846152    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846156    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846159    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846162    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846167    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846171    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846175    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846179    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846182    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846186    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846188    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846192    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846194    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846197    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846200    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846203    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.847728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846207    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846210    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846213    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846215    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846218    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846221    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846224    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846227    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846229    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846232    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846234    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846237    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846240    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846243    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846245    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846248    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846251    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846253    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846257    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846260    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.848234 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846262    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846265    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846268    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846271    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846274    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846277    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846279    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846282    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846285    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846289    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846292    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846294    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846297    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846301    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846303    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846306    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846308    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846311    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846314    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846317    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.848728 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846319    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846321    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846324    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846327    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846329    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846333    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846336    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846339    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846342    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846344    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846347    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846351    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846353    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846356    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846359    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846361    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846365    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846367    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846370    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846373    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.849252 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846375    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846379    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846382    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846384    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846387    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846390    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846392    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846395    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846398    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846406    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846409    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846411    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846414    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846417    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846420    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846422    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846425    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846428    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846431    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846814    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.849767 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846821    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846826    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846830    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846834    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846837    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846840    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846843    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846846    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846850    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846853    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846856    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846859    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846861    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846864    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846867    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846870    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846873    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846875    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846879    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846882    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.850306 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846885    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846888    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846890    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846893    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846896    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846899    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846902    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846904    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846907    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846909    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846912    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846914    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846917    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846919    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846922    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846925    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846928    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846931    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846933    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846936    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.850792 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846938    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846941    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846944    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846946    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846949    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846952    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846955    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846957    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846960    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846963    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846965    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846968    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846971    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846975    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846979    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846982    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846986    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846989    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846992    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.851297 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846995    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.846998    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847001    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847003    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847006    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847009    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847012    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847014    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847017    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847020    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847023    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847026    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847028    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847031    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847033    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847036    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847038    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847058    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847065    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847068    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.851798 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847070    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847075    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847078    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847081    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847084    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847087    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847089    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847092    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847095    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847097    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847100    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847103    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847106    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847109    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847111    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847114    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847117    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847120    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847122    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847206    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 29 21:16:23.852322 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847215    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847222    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847226    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847232    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847235    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847239    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847244    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847247    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847250    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847253    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847259    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847262    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847265    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847271    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847274    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847277    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847280    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847285    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847288    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847291    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847295    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847299    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847303    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 29 21:16:23.852808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847306    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847309    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847312    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847315    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847318    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847321    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847325    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847328    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847332    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847336    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847338    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847341    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847344    2565 flags.go:64] FLAG: --enable-server="true"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847348    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847354    2565 flags.go:64] FLAG: --event-burst="100"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847357    2565 flags.go:64] FLAG: --event-qps="50"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847360    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847363    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847367    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847371    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847374    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847377    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847382    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847385    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847388    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 29 21:16:23.853369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847391    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847394    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847397    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847400    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847403    2565 flags.go:64] FLAG: --feature-gates=""
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847407    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847409    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847412    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847416    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847419    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847422    2565 flags.go:64] FLAG: --help="false"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847425    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-123.ec2.internal"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847428    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847431    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847434    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847437    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847440    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847443    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847446    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847449    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847452    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847456    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847459    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847462    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 29 21:16:23.854000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847465    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847468    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847471    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847474    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847477    2565 flags.go:64] FLAG: --lock-file=""
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847482    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847485    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847488    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847493    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847496    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847499    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847502    2565 flags.go:64] FLAG: --logging-format="text"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847505    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847509    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847512    2565 flags.go:64] FLAG: --manifest-url=""
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847514    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847519    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847522    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847527    2565 flags.go:64] FLAG: --max-pods="110"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847530    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847533    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847536    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847539    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847543    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847546    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 29 21:16:23.854623 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847549    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847558    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847561    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847564    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847567    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847570    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847577    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847580    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847583    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847586    2565 flags.go:64] FLAG: --port="10250"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847589    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847592    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08be74d0d658ea2c6"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847597    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847600    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847604    2565 flags.go:64] FLAG: --register-node="true"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847607    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847611    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847614    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847617    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847620    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847623    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847626    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847630    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847633    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847636    2565 flags.go:64] FLAG: --runonce="false"
Jul 29 21:16:23.855368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847639    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847643    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847646    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847649    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847652    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847655    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847659    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847662    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847665    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847668    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847671    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847673    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847677    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847680    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847683    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847688    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847691    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847695    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847699    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847706    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847709    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847711    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847714    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847717    2565 flags.go:64] FLAG: --v="2"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847721    2565 flags.go:64] FLAG: --version="false"
Jul 29 21:16:23.855974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847726    2565 flags.go:64] FLAG: --vmodule=""
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847730    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.847733    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847850    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847856    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847860    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847863    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847865    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847868    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847871    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847874    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847876    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847879    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847882    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847884    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847887    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847890    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847892    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847895    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847897    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.856608 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847900    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847903    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847907    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847909    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847917    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847921    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847924    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847927    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847930    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847933    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847935    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847938    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847941    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847944    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847947    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847949    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847952    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847955    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847958    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.857118 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847961    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847964    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847968    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847971    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847974    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847977    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847980    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847983    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847986    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847989    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847993    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847996    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.847999    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848002    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848005    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848008    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848012    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848015    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848020    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.857855 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848022    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848025    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848028    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848030    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848033    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848036    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848038    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848057    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848060    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848062    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848065    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848068    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848071    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848073    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848076    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848078    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848081    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848084    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848087    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848089    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.858499 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848092    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848094    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848097    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848100    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848102    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848105    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848107    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848110    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848114    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848118    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848121    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848125    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848128    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848131    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848134    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848136    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848139    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848142    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848145    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.859029 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848148    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848150    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848153    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.848155    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.848901    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.856307    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.856328    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.856350    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.856360    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856422    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856428    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856431    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856434    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856437    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856440    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.859524 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856443    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856446    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856452    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856457    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856461    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856464    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856467    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856470    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856473    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856476    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856479    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856482    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856485    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856488    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856491    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856494    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856497    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856500    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.859907 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856503    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856506    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856509    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856512    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856514    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856519    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856522    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856525    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856528    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856531    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856533    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856536    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856539    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856541    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856544    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856547    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856549    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856552    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856555    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856557    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.860370 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856560    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856562    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856565    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856568    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856571    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856573    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856576    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856580    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856583    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856586    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856589    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856592    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856595    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856598    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856601    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856604    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856607    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856611    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856614    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856617    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.860848 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856620    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856623    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856626    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856629    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856632    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856634    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856637    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856640    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856643    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856645    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856648    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856651    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856653    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856656    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856659    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856662    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856664    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856667    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856670    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856672    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.861363 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856675    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856678    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856680    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856683    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856686    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856688    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856691    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856693    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856696    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856699    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856702    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856705    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856708    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856711    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856713    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.861862 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.856719    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856870    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856875    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856878    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856881    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856884    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856886    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856889    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856892    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856894    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856898    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856900    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856904    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856907    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856909    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856912    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856915    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856917    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856920    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856922    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 29 21:16:23.862272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856925    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856927    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856930    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856933    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856935    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856938    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856941    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856944    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856948    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856953    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856956    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856959    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856962    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856965    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856967    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856970    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856973    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856976    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856978    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 29 21:16:23.862755 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856981    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856983    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856986    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856988    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856991    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856994    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.856997    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857000    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857002    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857005    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857008    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857010    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857013    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857016    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857019    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857021    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857024    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857027    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857030    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857033    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 29 21:16:23.863312 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857036    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857038    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857054    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857057    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857060    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857063    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857066    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857068    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857071    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857075    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857079    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857083    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857085    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857088    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857091    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857094    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857096    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857099    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857102    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 29 21:16:23.863800 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857105    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857107    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857110    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857116    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857119    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857122    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857125    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857128    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857131    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857135    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857138    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857141    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857144    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857147    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857150    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857153    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857156    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857159    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857161    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 29 21:16:23.864275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857164    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:23.857167    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.857173    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.858806    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.861260    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.863014    2565 server.go:1019] "Starting client certificate rotation"
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.863145    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 21:16:23.864766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.863217    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 29 21:16:23.866082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.866067    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 21:16:23.867276 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.867254    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 29 21:16:23.881595 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.881574    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 29 21:16:23.884066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.884052    2565 log.go:25] "Validated CRI v1 image API"
Jul 29 21:16:23.884886 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.884870    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 29 21:16:23.887494 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.887466    2565 fs.go:135] Filesystem UUIDs: map[6ecb38bd-ec8d-4d48-8e99-2c802214706a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 dac94a10-10e2-4334-a6e7-d91db2de92ef:/dev/nvme0n1p3]
Jul 29 21:16:23.887578 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.887493    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 29 21:16:23.894131 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.894100    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 21:16:23.895508 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.895401    2565 manager.go:216] Machine: {Timestamp:2026-07-29 21:16:23.894863623 +0000 UTC m=+0.366442690 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099675 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2957e499aa71910865fb29c9806c11 SystemUUID:ec2957e4-99aa-7191-0865-fb29c9806c11 BootID:30dadf89-b43f-41ac-a5cf-4b4c3bfb89f4 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:87:c5:c4:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:87:c5:c4:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:b1:57:d0:91:5b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 29 21:16:23.895508 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.895505    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 29 21:16:23.895613 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.895597    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 29 21:16:23.899078 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899030    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 29 21:16:23.899221 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899083    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-123.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 29 21:16:23.899971 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899961    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 29 21:16:23.900006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899973    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 29 21:16:23.900006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899981    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 21:16:23.900006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.899992    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 29 21:16:23.901255 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.901244    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 29 21:16:23.904093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904081    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 21:16:23.904215 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904206    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 29 21:16:23.904251 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904237    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 29 21:16:23.904284 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904258    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 29 21:16:23.904284 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904273    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 29 21:16:23.904284 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904283    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 29 21:16:23.904369 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904293    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 29 21:16:23.904694 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904680    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 21:16:23.904749 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.904703    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 29 21:16:23.907083 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.907068    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 29 21:16:23.909706 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.909691    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 29 21:16:23.909764 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.909722    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 29 21:16:23.911719 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911707    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911723    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911730    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911737    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911745    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911753    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911766    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 29 21:16:23.911772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911773    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 29 21:16:23.911979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911781    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 29 21:16:23.911979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911787    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 29 21:16:23.911979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911796    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 29 21:16:23.911979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.911809    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 29 21:16:23.912765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.912750    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 29 21:16:23.912819 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.912772    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 29 21:16:23.913267 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.913240    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 29 21:16:23.913267 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.913242    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 29 21:16:23.914847 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.914827    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-123.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 29 21:16:23.916090 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.916076    2565 server.go:1268] "Started kubelet"
Jul 29 21:16:23.916203 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.916177    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 29 21:16:23.916335 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.916270    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 29 21:16:23.916886 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.916865    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 29 21:16:23.917183 ip-10-0-129-123 systemd[1]: Started Kubernetes Kubelet.
Jul 29 21:16:23.919223 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.919195    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 29 21:16:23.919808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.919796    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 29 21:16:23.922899 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.922879    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 29 21:16:23.924970 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.923810    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-123.ec2.internal.18c6df43a49453cc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-123.ec2.internal,UID:ip-10-0-129-123.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-123.ec2.internal,},FirstTimestamp:2026-07-29 21:16:23.915967436 +0000 UTC m=+0.387546568,LastTimestamp:2026-07-29 21:16:23.915967436 +0000 UTC m=+0.387546568,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-123.ec2.internal,}"
Jul 29 21:16:23.928756 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.928733    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 29 21:16:23.928858 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.928758    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 29 21:16:23.929792 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.929734    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 29 21:16:23.929984 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.929962    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 29 21:16:23.930068 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.929989    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 29 21:16:23.930068 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.930016    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:23.931295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931228    2565 factory.go:55] Registering systemd factory
Jul 29 21:16:23.931295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931246    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 29 21:16:23.931599 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931583    2565 factory.go:153] Registering CRI-O factory
Jul 29 21:16:23.931671 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931602    2565 factory.go:223] Registration of the crio container factory successfully
Jul 29 21:16:23.931730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931669    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 29 21:16:23.931730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931701    2565 factory.go:103] Registering Raw factory
Jul 29 21:16:23.931730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.931719    2565 manager.go:1196] Started watching for new ooms in manager
Jul 29 21:16:23.932218 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.932190    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 29 21:16:23.932340 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.932326    2565 manager.go:349] Starting recovery of all containers
Jul 29 21:16:23.932457 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:23.932431    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-123.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 29 21:16:23.935600 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.935580    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 29 21:16:23.935600 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.935599    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 29 21:16:23.939167 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.939119    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qk5mt"
Jul 29 21:16:23.943931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.943906    2565 manager.go:354] Recovery completed
Jul 29 21:16:23.947664 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.947642    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qk5mt"
Jul 29 21:16:23.948482 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.948468    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:23.951755 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.951732    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:23.951864 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.951770    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:23.951864 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.951786    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:23.952465 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.952448    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 29 21:16:23.952465 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.952462    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 29 21:16:23.952617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.952458    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:23.952617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.952482    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 29 21:16:23.953418 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.953398    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:23.953491 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.953431    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:23.953491 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.953446    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:23.955128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.955115    2565 policy_none.go:49] "None policy: Start"
Jul 29 21:16:23.955208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.955134    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 29 21:16:23.955208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.955148    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 29 21:16:23.957156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:23.957142    2565 policy_none.go:47] "Start"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.004766    2565 manager.go:337] "Starting Device Plugin manager"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.004797    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.004808    2565 server.go:87] "Starting device plugin registration server"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.005084    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.005097    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.005189    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.005329    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.005519    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.005950    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 29 21:16:24.014678 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.006002    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.047032 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.046986    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 29 21:16:24.048244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.048226    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 29 21:16:24.048318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.048266    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 29 21:16:24.048318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.048292    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 29 21:16:24.048422 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.048334    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 29 21:16:24.050589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.050568    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:24.105512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.105427    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:24.106464 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.106444    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:24.106571 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.106482    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:24.106571 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.106497    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:24.106571 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.106521    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.115673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.115657    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.115723 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.115682    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-123.ec2.internal\": node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.142062 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.142025    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.149015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.148988    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"]
Jul 29 21:16:24.149111 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.149067    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:24.150597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.150580    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:24.150683 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.150613    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:24.150683 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.150626    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:24.153113 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153100    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:24.153280 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.153337 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153294    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:24.153927 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153907    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:24.154002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153938    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:24.154002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153959    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:24.154002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.153908    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:24.154134 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.154009    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:24.154134 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.154033    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:24.156323 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.156305    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.156402 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.156334    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 29 21:16:24.157249 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.157231    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientMemory"
Jul 29 21:16:24.157336 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.157255    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasNoDiskPressure"
Jul 29 21:16:24.157336 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.157265    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeHasSufficientPID"
Jul 29 21:16:24.174781 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.174760    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-123.ec2.internal\" not found" node="ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.180458 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.180436    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-123.ec2.internal\" not found" node="ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.237021 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.236985    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc048883d1237889be55fccbd385befa-config\") pod \"kube-apiserver-proxy-ip-10-0-129-123.ec2.internal\" (UID: \"fc048883d1237889be55fccbd385befa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.237021 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.237019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.237247 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.237037    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.242509 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.242493    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.337916 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.337916 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.338132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc048883d1237889be55fccbd385befa-config\") pod \"kube-apiserver-proxy-ip-10-0-129-123.ec2.internal\" (UID: \"fc048883d1237889be55fccbd385befa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.338132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.338132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba3c82d5fadfae03b88bcec511ea7084-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal\" (UID: \"ba3c82d5fadfae03b88bcec511ea7084\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.338132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.337998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc048883d1237889be55fccbd385befa-config\") pod \"kube-apiserver-proxy-ip-10-0-129-123.ec2.internal\" (UID: \"fc048883d1237889be55fccbd385befa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.343007 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.342980    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.443784 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.443719    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.477932 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.477897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.483771 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.483754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.544239 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.544212    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.645225 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.645185    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.745702 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.745620    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.816971 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.816935    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:24.846651 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:24.846619    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-123.ec2.internal\" not found"
Jul 29 21:16:24.862437 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.862404    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 29 21:16:24.862583 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.862556    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 21:16:24.862647 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.862589    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 29 21:16:24.929960 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.929918    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:24.930931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.930910    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.931119 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.931102    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:24.954208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.954167    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 21:11:24 +0000 UTC" deadline="2028-01-31 12:20:14.692412516 +0000 UTC"
Jul 29 21:16:24.954208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.954199    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13215h3m49.738218355s"
Jul 29 21:16:24.954505 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.954488    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 21:16:24.955360 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.955345    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal"
Jul 29 21:16:24.967531 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:24.967507    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 29 21:16:25.117199 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:25.117160    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3c82d5fadfae03b88bcec511ea7084.slice/crio-50888f484d92b9656829a1b72cd529c94446e9a639478a039f2f8a6c00166b75 WatchSource:0}: Error finding container 50888f484d92b9656829a1b72cd529c94446e9a639478a039f2f8a6c00166b75: Status 404 returned error can't find the container with id 50888f484d92b9656829a1b72cd529c94446e9a639478a039f2f8a6c00166b75
Jul 29 21:16:25.126554 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.126529    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:16:25.133877 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:25.133853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc048883d1237889be55fccbd385befa.slice/crio-9236bc05419268cdaa6a7d6e18ea209c78f13400197b8e44454049818d967829 WatchSource:0}: Error finding container 9236bc05419268cdaa6a7d6e18ea209c78f13400197b8e44454049818d967829: Status 404 returned error can't find the container with id 9236bc05419268cdaa6a7d6e18ea209c78f13400197b8e44454049818d967829
Jul 29 21:16:25.906074 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.906022    2565 apiserver.go:52] "Watching apiserver"
Jul 29 21:16:25.913118 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.912872    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 29 21:16:25.916204 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.916169    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8ktwf","kube-system/konnectivity-agent-z9scg","kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal","openshift-image-registry/node-ca-ljfcj","openshift-multus/network-metrics-daemon-cf85j","openshift-network-diagnostics/network-check-target-pm5pq","openshift-network-operator/iptables-alerter-4hsfw","kube-system/global-pull-secret-syncer-57rmr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx","openshift-cluster-node-tuning-operator/tuned-cf6hq","openshift-dns/node-resolver-xgrr6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal","openshift-multus/multus-additional-cni-plugins-fz857","openshift-multus/multus-s5f4b"]
Jul 29 21:16:25.919082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.919041    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.921414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.921390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.922145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.922125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.922250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.922125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.922250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.922214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l4np9\""
Jul 29 21:16:25.923769 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.923741    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lt4vl\""
Jul 29 21:16:25.924088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.924059    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 29 21:16:25.924343 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.924323    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.924587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.924556    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.924788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.924771    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 29 21:16:25.924972 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.924958    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 29 21:16:25.925753 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.925734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.926715 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.926695    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:25.931229 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.931205    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 29 21:16:25.932748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.931795    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:25.932748 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:25.931888    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:25.932748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.932101    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:25.932748 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:25.932175    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:25.932748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.932587    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 29 21:16:25.933997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.933505    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 29 21:16:25.933997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.933701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.933997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.933701    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2l4n6\""
Jul 29 21:16:25.933997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.933783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdvf6\""
Jul 29 21:16:25.934285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.934151    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.935983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.935961    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:25.938351 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.938327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.938485 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.938467    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.938626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.938608    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 29 21:16:25.938813 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.938795    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9dlx5\""
Jul 29 21:16:25.938889 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.938878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:25.941779 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.941742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:25.943561 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.943544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.943712 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.943614    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-452fh\""
Jul 29 21:16:25.943777 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.943728    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.943998 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.943945    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hk8sh\""
Jul 29 21:16:25.944070 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.944001    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 29 21:16:25.944341 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.944323    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:25.944660 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.944640    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 29 21:16:25.947490 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.947116    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:25.947835 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.947815    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8h4d5\""
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-host\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-tmp\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hz5t\" (UniqueName: \"kubernetes.io/projected/f7f02b40-5c6f-489f-90ba-3443f4845c52-kube-api-access-6hz5t\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-kubelet\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-multus-certs\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948302    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c25xc\" (UniqueName: \"kubernetes.io/projected/76a8f833-1ec6-459f-a42f-23f8f6211ed6-kube-api-access-c25xc\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-conf\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.948447 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948444    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-var-lib-kubelet\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948528    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-system-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-kubernetes\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-cnibin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-os-release\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-cni-binary-copy\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948792    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-k8s-cni-cncf-io\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-netns\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.948995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysconfig\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-systemd\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949033    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-sys\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:25.949548    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949842    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-conf-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-lib-modules\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.949915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-socket-dir-parent\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.949968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-bin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-hostroot\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-etc-kubernetes\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1a702f0-7382-4c43-b759-1466980f155e-serviceca\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-run\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-os-release\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqdc\" (UniqueName: \"kubernetes.io/projected/0f21c097-c155-4334-b084-b8b094f0cac8-kube-api-access-8sqdc\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-modprobe-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-tuned\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-cnibin\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.950426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrqm\" (UniqueName: \"kubernetes.io/projected/a6243582-85a5-456a-8acb-a76632024c98-kube-api-access-vzrqm\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-multus\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950493    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1a702f0-7382-4c43-b759-1466980f155e-host\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-system-cni-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-multus-daemon-config\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950612    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28k5h\" (UniqueName: \"kubernetes.io/projected/c1a702f0-7382-4c43-b759-1466980f155e-kube-api-access-28k5h\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:25.951088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.950889    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x6v47\""
Jul 29 21:16:25.951461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.951094    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 29 21:16:25.951461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.951306    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 29 21:16:25.951461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.951437    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 29 21:16:25.983009 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.982976    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqgbn"
Jul 29 21:16:25.993309 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:25.993140    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kqgbn"
Jul 29 21:16:26.030844 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.030813    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 29 21:16:26.050852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-run\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050854    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-os-release\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqdc\" (UniqueName: \"kubernetes.io/projected/0f21c097-c155-4334-b084-b8b094f0cac8-kube-api-access-8sqdc\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-sys-fs\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050945    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-modprobe-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-tuned\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.050993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-cnibin\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.051029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1a702f0-7382-4c43-b759-1466980f155e-host\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-kubelet\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-slash\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f9ce67f-8d99-4397-8fe6-395e53f23440-hosts-file\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6233883-03f0-4ab6-b06d-d895a418213d-iptables-alerter-script\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-multus-daemon-config\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28k5h\" (UniqueName: \"kubernetes.io/projected/c1a702f0-7382-4c43-b759-1466980f155e-kube-api-access-28k5h\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9ce67f-8d99-4397-8fe6-395e53f23440-tmp-dir\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnv4w\" (UniqueName: \"kubernetes.io/projected/e6233883-03f0-4ab6-b06d-d895a418213d-kube-api-access-rnv4w\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-dbus\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.051417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-host\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051440    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-os-release\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-cnibin\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051478    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-modprobe-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051498    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-run\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-tmp\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-host\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hz5t\" (UniqueName: \"kubernetes.io/projected/f7f02b40-5c6f-489f-90ba-3443f4845c52-kube-api-access-6hz5t\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-multus-certs\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1a702f0-7382-4c43-b759-1466980f155e-host\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-multus-certs\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051747    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-log-socket\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d54df4e4-1468-4fdf-91cf-2ad540cacf25-konnectivity-ca\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.051949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.051875    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-var-lib-kubelet\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052039    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-system-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-os-release\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-config\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-multus-daemon-config\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052201    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-cnibin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-var-lib-kubelet\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-cni-binary-copy\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-k8s-cni-cncf-io\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-cnibin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052295    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-system-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-os-release\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-systemd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-k8s-cni-cncf-io\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-env-overrides\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.052695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-registration-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052507    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052550    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntlh\" (UniqueName: \"kubernetes.io/projected/826e07a4-4116-4f2e-8000-53324d87c284-kube-api-access-hntlh\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052582    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-socket-dir-parent\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052611    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1a702f0-7382-4c43-b759-1466980f155e-serviceca\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052668    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052694    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvm4s\" (UniqueName: \"kubernetes.io/projected/f6622c9a-60c2-4e17-aaf6-f98042020807-kube-api-access-cvm4s\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-netns\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-socket-dir-parent\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-var-lib-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-etc-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0f21c097-c155-4334-b084-b8b094f0cac8-cni-binary-copy\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-bin\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.053551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrqm\" (UniqueName: \"kubernetes.io/projected/a6243582-85a5-456a-8acb-a76632024c98-kube-api-access-vzrqm\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-multus\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-systemd-units\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052942    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-system-cni-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-d\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.052966    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6233883-03f0-4ab6-b06d-d895a418213d-host-slash\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053021    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-multus\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c1a702f0-7382-4c43-b759-1466980f155e-serviceca\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-kubelet\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-system-cni-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-script-lib\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-kubelet\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qgl7\" (UniqueName: \"kubernetes.io/projected/8f9ce67f-8d99-4397-8fe6-395e53f23440-kube-api-access-4qgl7\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c25xc\" (UniqueName: \"kubernetes.io/projected/76a8f833-1ec6-459f-a42f-23f8f6211ed6-kube-api-access-c25xc\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-conf\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053261    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.054390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-netns\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-cni-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-bin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysctl-conf\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053442    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-hostroot\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-run-netns\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-kubernetes\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-host-var-lib-cni-bin\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-conf-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-kubernetes\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053591    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d54df4e4-1468-4fdf-91cf-2ad540cacf25-agent-certs\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053611    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-hostroot\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-ovn\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-node-log\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.053703    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:26.055244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysconfig\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-multus-conf-dir\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.053793    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:26.55375728 +0000 UTC m=+3.025336338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-sysconfig\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053853    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-systemd\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-sys\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-systemd\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053963    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-sys\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.053995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6243582-85a5-456a-8acb-a76632024c98-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-netd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-socket-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-kubelet-config\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-lib-modules\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056047 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-etc-kubernetes\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/826e07a4-4116-4f2e-8000-53324d87c284-ovn-node-metrics-cert\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-device-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f7f02b40-5c6f-489f-90ba-3443f4845c52-lib-modules\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal" event={"ID":"fc048883d1237889be55fccbd385befa","Type":"ContainerStarted","Data":"9236bc05419268cdaa6a7d6e18ea209c78f13400197b8e44454049818d967829"}
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054353    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f21c097-c155-4334-b084-b8b094f0cac8-etc-kubernetes\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.054743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6243582-85a5-456a-8acb-a76632024c98-cni-binary-copy\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.055745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-etc-tuned\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.056677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.055790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f02b40-5c6f-489f-90ba-3443f4845c52-tmp\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.057081 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.056733    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal" event={"ID":"ba3c82d5fadfae03b88bcec511ea7084","Type":"ContainerStarted","Data":"50888f484d92b9656829a1b72cd529c94446e9a639478a039f2f8a6c00166b75"}
Jul 29 21:16:26.058108 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.058084    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:26.058277 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.058114    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:26.058277 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.058143    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:26.058277 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.058231    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:26.558211458 +0000 UTC m=+3.029790520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:26.061320 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.061300    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28k5h\" (UniqueName: \"kubernetes.io/projected/c1a702f0-7382-4c43-b759-1466980f155e-kube-api-access-28k5h\") pod \"node-ca-ljfcj\" (UID: \"c1a702f0-7382-4c43-b759-1466980f155e\") " pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.062260 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.062238    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hz5t\" (UniqueName: \"kubernetes.io/projected/f7f02b40-5c6f-489f-90ba-3443f4845c52-kube-api-access-6hz5t\") pod \"tuned-cf6hq\" (UID: \"f7f02b40-5c6f-489f-90ba-3443f4845c52\") " pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.062389 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.062274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqdc\" (UniqueName: \"kubernetes.io/projected/0f21c097-c155-4334-b084-b8b094f0cac8-kube-api-access-8sqdc\") pod \"multus-s5f4b\" (UID: \"0f21c097-c155-4334-b084-b8b094f0cac8\") " pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.064074 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.064046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c25xc\" (UniqueName: \"kubernetes.io/projected/76a8f833-1ec6-459f-a42f-23f8f6211ed6-kube-api-access-c25xc\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:26.064361 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.064339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrqm\" (UniqueName: \"kubernetes.io/projected/a6243582-85a5-456a-8acb-a76632024c98-kube-api-access-vzrqm\") pod \"multus-additional-cni-plugins-fz857\" (UID: \"a6243582-85a5-456a-8acb-a76632024c98\") " pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.154616 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-log-socket\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d54df4e4-1468-4fdf-91cf-2ad540cacf25-konnectivity-ca\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.154807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-log-socket\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-config\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-systemd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-env-overrides\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154890    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-registration-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154928    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-systemd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.154983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntlh\" (UniqueName: \"kubernetes.io/projected/826e07a4-4116-4f2e-8000-53324d87c284-kube-api-access-hntlh\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.154975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvm4s\" (UniqueName: \"kubernetes.io/projected/f6622c9a-60c2-4e17-aaf6-f98042020807-kube-api-access-cvm4s\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-netns\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-var-lib-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-etc-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-bin\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-systemd-units\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6233883-03f0-4ab6-b06d-d895a418213d-host-slash\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-script-lib\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d54df4e4-1468-4fdf-91cf-2ad540cacf25-konnectivity-ca\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qgl7\" (UniqueName: \"kubernetes.io/projected/8f9ce67f-8d99-4397-8fe6-395e53f23440-kube-api-access-4qgl7\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-bin\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.155345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155335    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-etc-selinux\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-systemd-units\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-config\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155430    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-var-lib-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e6233883-03f0-4ab6-b06d-d895a418213d-host-slash\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-netns\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-registration-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-etc-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155670    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-openvswitch\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155673    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-env-overrides\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d54df4e4-1468-4fdf-91cf-2ad540cacf25-agent-certs\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155740    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-ovn\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155763    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-node-log\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-run-ovn\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-node-log\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155925    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.155959    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155966    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-netd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-socket-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.155999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-kubelet-config\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.156027    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:26.656006044 +0000 UTC m=+3.127585115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-kubelet-config\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156059    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/826e07a4-4116-4f2e-8000-53324d87c284-ovn-node-metrics-cert\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-device-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-cni-netd\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-sys-fs\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156154    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/826e07a4-4116-4f2e-8000-53324d87c284-ovnkube-script-lib\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156166    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-device-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-kubelet\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-sys-fs\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.156935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-slash\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-kubelet\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f9ce67f-8d99-4397-8fe6-395e53f23440-hosts-file\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/826e07a4-4116-4f2e-8000-53324d87c284-host-slash\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6233883-03f0-4ab6-b06d-d895a418213d-iptables-alerter-script\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9ce67f-8d99-4397-8fe6-395e53f23440-tmp-dir\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f9ce67f-8d99-4397-8fe6-395e53f23440-hosts-file\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnv4w\" (UniqueName: \"kubernetes.io/projected/e6233883-03f0-4ab6-b06d-d895a418213d-kube-api-access-rnv4w\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-dbus\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156482    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f78c6650-9271-48e4-b274-cfd7e4d542b3-dbus\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f9ce67f-8d99-4397-8fe6-395e53f23440-tmp-dir\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156862    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f6622c9a-60c2-4e17-aaf6-f98042020807-socket-dir\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.157503 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.156906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e6233883-03f0-4ab6-b06d-d895a418213d-iptables-alerter-script\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.158261 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.158240    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d54df4e4-1468-4fdf-91cf-2ad540cacf25-agent-certs\") pod \"konnectivity-agent-z9scg\" (UID: \"d54df4e4-1468-4fdf-91cf-2ad540cacf25\") " pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.158765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.158697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/826e07a4-4116-4f2e-8000-53324d87c284-ovn-node-metrics-cert\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.166535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.166510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntlh\" (UniqueName: \"kubernetes.io/projected/826e07a4-4116-4f2e-8000-53324d87c284-kube-api-access-hntlh\") pod \"ovnkube-node-8ktwf\" (UID: \"826e07a4-4116-4f2e-8000-53324d87c284\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.166705 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.166687    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qgl7\" (UniqueName: \"kubernetes.io/projected/8f9ce67f-8d99-4397-8fe6-395e53f23440-kube-api-access-4qgl7\") pod \"node-resolver-xgrr6\" (UID: \"8f9ce67f-8d99-4397-8fe6-395e53f23440\") " pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.167031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.167014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvm4s\" (UniqueName: \"kubernetes.io/projected/f6622c9a-60c2-4e17-aaf6-f98042020807-kube-api-access-cvm4s\") pod \"aws-ebs-csi-driver-node-wrggx\" (UID: \"f6622c9a-60c2-4e17-aaf6-f98042020807\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.170309 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.170285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnv4w\" (UniqueName: \"kubernetes.io/projected/e6233883-03f0-4ab6-b06d-d895a418213d-kube-api-access-rnv4w\") pod \"iptables-alerter-4hsfw\" (UID: \"e6233883-03f0-4ab6-b06d-d895a418213d\") " pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.236129 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.236088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-cf6hq"
Jul 29 21:16:26.253351 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.253322    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fz857"
Jul 29 21:16:26.258577 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.258553    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s5f4b"
Jul 29 21:16:26.267465 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.267435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ljfcj"
Jul 29 21:16:26.276842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.276819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx"
Jul 29 21:16:26.286597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.286578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xgrr6"
Jul 29 21:16:26.294647 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.294625    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:26.295788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.295767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:26.310783 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.310757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:26.318117 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.318085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4hsfw"
Jul 29 21:16:26.560390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.560296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:26.560390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.560375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:26.560610 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560473    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:26.560610 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560490    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:26.560610 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560509    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:26.560610 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560519    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:26.560610 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560562    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.56053635 +0000 UTC m=+4.032115428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:26.560825 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.560616    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.560598432 +0000 UTC m=+4.032177490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:26.661806 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.661764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:26.661972 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.661943    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:26.662089 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:26.662022    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.662006794 +0000 UTC m=+4.133585849 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:26.843823 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.843751    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 29 21:16:26.996351 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.996294    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 21:11:26 +0000 UTC" deadline="2028-01-05 23:21:58.055600113 +0000 UTC"
Jul 29 21:16:26.996351 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:26.996345    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12602h5m31.059257854s"
Jul 29 21:16:27.049215 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.049177    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:27.049403 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.049320    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:27.049403 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.049372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:27.049522 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.049489    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:27.222513 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.222485    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f21c097_c155_4334_b084_b8b094f0cac8.slice/crio-02ce55e451aa8643367ac480420028efc225766e725b1785236cfec37dacdd86 WatchSource:0}: Error finding container 02ce55e451aa8643367ac480420028efc225766e725b1785236cfec37dacdd86: Status 404 returned error can't find the container with id 02ce55e451aa8643367ac480420028efc225766e725b1785236cfec37dacdd86
Jul 29 21:16:27.223906 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.223880    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9ce67f_8d99_4397_8fe6_395e53f23440.slice/crio-6cdb414e9ad3e6c0b6fca77febf8afaaa42b9be759f09179deee444db104eae8 WatchSource:0}: Error finding container 6cdb414e9ad3e6c0b6fca77febf8afaaa42b9be759f09179deee444db104eae8: Status 404 returned error can't find the container with id 6cdb414e9ad3e6c0b6fca77febf8afaaa42b9be759f09179deee444db104eae8
Jul 29 21:16:27.245905 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.245871    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod826e07a4_4116_4f2e_8000_53324d87c284.slice/crio-fb781913c354f524f96454d21efd4faa4c819c496214713bf09adb62e03593a7 WatchSource:0}: Error finding container fb781913c354f524f96454d21efd4faa4c819c496214713bf09adb62e03593a7: Status 404 returned error can't find the container with id fb781913c354f524f96454d21efd4faa4c819c496214713bf09adb62e03593a7
Jul 29 21:16:27.247570 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.247466    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f02b40_5c6f_489f_90ba_3443f4845c52.slice/crio-4efd78a7605b80fd5ad51b5fa509f1a2baf22d82a12654a93b9ee6ffc638b12b WatchSource:0}: Error finding container 4efd78a7605b80fd5ad51b5fa509f1a2baf22d82a12654a93b9ee6ffc638b12b: Status 404 returned error can't find the container with id 4efd78a7605b80fd5ad51b5fa509f1a2baf22d82a12654a93b9ee6ffc638b12b
Jul 29 21:16:27.254016 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.253979    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6233883_03f0_4ab6_b06d_d895a418213d.slice/crio-3a6c2ddcdf5f5905de349d671e591d1cece64eb86966a11bf6e0e5252abd6c33 WatchSource:0}: Error finding container 3a6c2ddcdf5f5905de349d671e591d1cece64eb86966a11bf6e0e5252abd6c33: Status 404 returned error can't find the container with id 3a6c2ddcdf5f5905de349d671e591d1cece64eb86966a11bf6e0e5252abd6c33
Jul 29 21:16:27.255070 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.255018    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54df4e4_1468_4fdf_91cf_2ad540cacf25.slice/crio-e81897d6ee38eb993ac00af0c30b58da10b36c8ffc6e940646a0d3df8ccb4b8a WatchSource:0}: Error finding container e81897d6ee38eb993ac00af0c30b58da10b36c8ffc6e940646a0d3df8ccb4b8a: Status 404 returned error can't find the container with id e81897d6ee38eb993ac00af0c30b58da10b36c8ffc6e940646a0d3df8ccb4b8a
Jul 29 21:16:27.256208 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.255922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6243582_85a5_456a_8acb_a76632024c98.slice/crio-a19bf18b8864707cf71154e2733ef4944773f2c303fdea544f54498a84acc8f9 WatchSource:0}: Error finding container a19bf18b8864707cf71154e2733ef4944773f2c303fdea544f54498a84acc8f9: Status 404 returned error can't find the container with id a19bf18b8864707cf71154e2733ef4944773f2c303fdea544f54498a84acc8f9
Jul 29 21:16:27.256999 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.256958    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a702f0_7382_4c43_b759_1466980f155e.slice/crio-5fef4e743235a2e5d76714ff779cc857c93a2961eefd8602a39c8e99aaf2105a WatchSource:0}: Error finding container 5fef4e743235a2e5d76714ff779cc857c93a2961eefd8602a39c8e99aaf2105a: Status 404 returned error can't find the container with id 5fef4e743235a2e5d76714ff779cc857c93a2961eefd8602a39c8e99aaf2105a
Jul 29 21:16:27.257880 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:16:27.257856    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6622c9a_60c2_4e17_aaf6_f98042020807.slice/crio-a6b1ae6f525dbe71a8f289b9cbae85a01afb979ad8c8b81b4ff69088f53b960a WatchSource:0}: Error finding container a6b1ae6f525dbe71a8f289b9cbae85a01afb979ad8c8b81b4ff69088f53b960a: Status 404 returned error can't find the container with id a6b1ae6f525dbe71a8f289b9cbae85a01afb979ad8c8b81b4ff69088f53b960a
Jul 29 21:16:27.568740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.568549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:27.568740 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568727    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568749    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568759    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.568770    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568811    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:29.568797483 +0000 UTC m=+6.040376539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568868    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:27.568939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.568916    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:29.568902709 +0000 UTC m=+6.040481764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:27.670171 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.670132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:27.670369 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.670315    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:27.670433 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:27.670395    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:29.670374076 +0000 UTC m=+6.141953154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:27.996793 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.996746    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 21:11:26 +0000 UTC" deadline="2028-03-19 17:54:35.284535864 +0000 UTC"
Jul 29 21:16:27.996793 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:27.996788    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14372h38m7.287752925s"
Jul 29 21:16:28.050309 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.049693    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:28.050309 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:28.049849    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:28.074527 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.074485    2565 generic.go:356] "Generic (PLEG): container finished" podID="ba3c82d5fadfae03b88bcec511ea7084" containerID="6014890bd00aacfdade462a505fbae3ad043f273469699f11c2f046a1ec67f87" exitCode=0
Jul 29 21:16:28.074758 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.074586    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal" event={"ID":"ba3c82d5fadfae03b88bcec511ea7084","Type":"ContainerDied","Data":"6014890bd00aacfdade462a505fbae3ad043f273469699f11c2f046a1ec67f87"}
Jul 29 21:16:28.089341 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.089265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ljfcj" event={"ID":"c1a702f0-7382-4c43-b759-1466980f155e","Type":"ContainerStarted","Data":"5fef4e743235a2e5d76714ff779cc857c93a2961eefd8602a39c8e99aaf2105a"}
Jul 29 21:16:28.097983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.097902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hsfw" event={"ID":"e6233883-03f0-4ab6-b06d-d895a418213d","Type":"ContainerStarted","Data":"3a6c2ddcdf5f5905de349d671e591d1cece64eb86966a11bf6e0e5252abd6c33"}
Jul 29 21:16:28.100315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.100289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgrr6" event={"ID":"8f9ce67f-8d99-4397-8fe6-395e53f23440","Type":"ContainerStarted","Data":"6cdb414e9ad3e6c0b6fca77febf8afaaa42b9be759f09179deee444db104eae8"}
Jul 29 21:16:28.103506 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.103443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5f4b" event={"ID":"0f21c097-c155-4334-b084-b8b094f0cac8","Type":"ContainerStarted","Data":"02ce55e451aa8643367ac480420028efc225766e725b1785236cfec37dacdd86"}
Jul 29 21:16:28.117276 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.117237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal" event={"ID":"fc048883d1237889be55fccbd385befa","Type":"ContainerStarted","Data":"1a8134349bb8a3694063b11f03985fa36b0fda40495572a7669bd3b2e19d9f8d"}
Jul 29 21:16:28.127666 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.127563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx" event={"ID":"f6622c9a-60c2-4e17-aaf6-f98042020807","Type":"ContainerStarted","Data":"a6b1ae6f525dbe71a8f289b9cbae85a01afb979ad8c8b81b4ff69088f53b960a"}
Jul 29 21:16:28.139759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.139718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerStarted","Data":"a19bf18b8864707cf71154e2733ef4944773f2c303fdea544f54498a84acc8f9"}
Jul 29 21:16:28.153149 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.153100    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9scg" event={"ID":"d54df4e4-1468-4fdf-91cf-2ad540cacf25","Type":"ContainerStarted","Data":"e81897d6ee38eb993ac00af0c30b58da10b36c8ffc6e940646a0d3df8ccb4b8a"}
Jul 29 21:16:28.156313 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.156277    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cf6hq" event={"ID":"f7f02b40-5c6f-489f-90ba-3443f4845c52","Type":"ContainerStarted","Data":"4efd78a7605b80fd5ad51b5fa509f1a2baf22d82a12654a93b9ee6ffc638b12b"}
Jul 29 21:16:28.163468 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:28.163436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"fb781913c354f524f96454d21efd4faa4c819c496214713bf09adb62e03593a7"}
Jul 29 21:16:29.050161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.049372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:29.050161 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.049518    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:29.050161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.049948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:29.050161 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.050044    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:29.178848 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.178808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal" event={"ID":"ba3c82d5fadfae03b88bcec511ea7084","Type":"ContainerStarted","Data":"497b0f8d9bafcf1dff2e96584d863f93d44c6540eb9dd26fd8afc77268b9db45"}
Jul 29 21:16:29.198184 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.198122    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-123.ec2.internal" podStartSLOduration=4.198099956 podStartE2EDuration="4.198099956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:29.198032259 +0000 UTC m=+5.669611341" watchObservedRunningTime="2026-07-29 21:16:29.198099956 +0000 UTC m=+5.669679039"
Jul 29 21:16:29.198738 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.198698    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-123.ec2.internal" podStartSLOduration=4.198688909 podStartE2EDuration="4.198688909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:28.134369226 +0000 UTC m=+4.605948309" watchObservedRunningTime="2026-07-29 21:16:29.198688909 +0000 UTC m=+5.670267988"
Jul 29 21:16:29.588344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.588307    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.588379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588447    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588466    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588472    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588485    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588528    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:33.588508937 +0000 UTC m=+10.060088009 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:29.588646 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.588547    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:33.588537036 +0000 UTC m=+10.060116093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:29.689625 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:29.689588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:29.689835 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.689762    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:29.689835 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:29.689821    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:33.689807331 +0000 UTC m=+10.161386389 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:30.051388 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:30.051301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:30.051888 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:30.051462    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:31.049434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:31.049394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:31.049620 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:31.049554    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:31.050174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:31.050148    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:31.050287 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:31.050265    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:32.049343 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:32.049311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:32.049805 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:32.049470    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:33.048807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:33.048768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:33.048988 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.048901    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:33.049096 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:33.048770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:33.049199 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.049178    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:33.622721 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:33.622682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:33.622739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622828    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622849    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622871    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622884    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622900    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:41.622880356 +0000 UTC m=+18.094459414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:33.623191 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.622931    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:41.622915995 +0000 UTC m=+18.094495054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:33.723794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:33.723754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:33.723989 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.723903    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:33.723989 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:33.723974    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:41.723954722 +0000 UTC m=+18.195533798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:34.049761 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:34.049681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:34.049920 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:34.049780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:35.048639 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:35.048607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:35.049123 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:35.048732    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:35.049123 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:35.048920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:35.049123 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:35.049009    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:36.049121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:36.049089    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:36.049556 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:36.049228    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:37.049176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:37.049137    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:37.049636 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:37.049187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:37.049636 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:37.049266    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:37.049636 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:37.049402    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:38.049375 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:38.049316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:38.049867 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:38.049476    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:39.048999 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:39.048962    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:39.049236 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:39.048959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:39.049236 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:39.049096    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:39.049236 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:39.049162    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:40.049457 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:40.049421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:40.049892 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:40.049570    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:41.049297 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:41.049259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:41.049526 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.049401    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:41.049526 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:41.049442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:41.049526 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.049516    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:41.680858 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:41.680803    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:41.680902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.680978    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.681026    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.681061    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.681069    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:57.681032742 +0000 UTC m=+34.152611808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.681077    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:41.681124 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.681128    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:57.68111283 +0000 UTC m=+34.152691908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:41.782162 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:41.782125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:41.782373 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.782271    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:41.782373 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:41.782362    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:57.782338633 +0000 UTC m=+34.253917707 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:42.049359 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:42.049284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:42.049478 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:42.049408    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:43.048852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:43.048814    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:43.049300 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:43.048831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:43.049300 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:43.048945    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:43.049300 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:43.049066    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:44.050104 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:44.050068    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:44.050548 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:44.050208    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:45.048839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:45.048803    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:45.049014 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:45.048805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:45.049014 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:45.048919    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:45.049161 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:45.049033    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:46.049533 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.049496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:46.049933 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:46.049646    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:46.228640 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.228250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z9scg" event={"ID":"d54df4e4-1468-4fdf-91cf-2ad540cacf25","Type":"ContainerStarted","Data":"3c39266092b146bad939ad1dcdb3b50bd4a79dbc45ba7dd1ce9b007d579068d8"}
Jul 29 21:16:46.230558 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.229985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-cf6hq" event={"ID":"f7f02b40-5c6f-489f-90ba-3443f4845c52","Type":"ContainerStarted","Data":"7b8f48a0f17750911a1862b079a19f41091dd81dec542729dc4066e1a093c157"}
Jul 29 21:16:46.232987 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.232839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"698668a4becd4f71af9e24ca15f4f6b48ef07b053291b7a04f850a6b9fabfa1c"}
Jul 29 21:16:46.232987 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.232925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"b0e44723e960da5143fc81f12ce2e22350089b16e6d6e2073fa690f7f4603f03"}
Jul 29 21:16:46.235380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.235148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ljfcj" event={"ID":"c1a702f0-7382-4c43-b759-1466980f155e","Type":"ContainerStarted","Data":"d89fbd1ff86139e65522691559a2878f20ccb5f7c5c61a4baf56211db89ccd72"}
Jul 29 21:16:46.237533 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.237479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xgrr6" event={"ID":"8f9ce67f-8d99-4397-8fe6-395e53f23440","Type":"ContainerStarted","Data":"6e6659ca1ec63e1d46e5142e9ba897bf705bd58e020493e9ad89007813be933a"}
Jul 29 21:16:46.239031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.238999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s5f4b" event={"ID":"0f21c097-c155-4334-b084-b8b094f0cac8","Type":"ContainerStarted","Data":"2cf5a76427cad6224ec54b13898d4194e215fd34826d31edb71e1a3d95cbed6e"}
Jul 29 21:16:46.240707 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.240560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx" event={"ID":"f6622c9a-60c2-4e17-aaf6-f98042020807","Type":"ContainerStarted","Data":"a4b59f04e757c92b5a9c3fd4dfff7a9551154ab0529053ae21bf99ba80ec4a62"}
Jul 29 21:16:46.244076 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.244011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerStarted","Data":"d5f4bc8a55fb1d05b07503f1ac9591e9a6dcce61b2e4f751101f75fb890ad6d4"}
Jul 29 21:16:46.246977 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.246875    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z9scg" podStartSLOduration=11.561714715 podStartE2EDuration="22.246856556s" totalImagesPullingTime="10.685141841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:46.245662613 +0000 UTC m=+22.717241691" watchObservedRunningTime="2026-07-29 21:16:46.246856556 +0000 UTC m=+22.718435634"
Jul 29 21:16:46.278643 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.278584    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-cf6hq" podStartSLOduration=3.5323401690000003 podStartE2EDuration="22.278567081s" totalImagesPullingTime="18.746226912s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:46.278534994 +0000 UTC m=+22.750114095" watchObservedRunningTime="2026-07-29 21:16:46.278567081 +0000 UTC m=+22.750146156"
Jul 29 21:16:46.278813 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.278704    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xgrr6" podStartSLOduration=3.565281896 podStartE2EDuration="22.278697366s" totalImagesPullingTime="18.71341547s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:46.260929761 +0000 UTC m=+22.732508862" watchObservedRunningTime="2026-07-29 21:16:46.278697366 +0000 UTC m=+22.750276444"
Jul 29 21:16:46.339396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:46.339340    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s5f4b" podStartSLOduration=3.528232129 podStartE2EDuration="22.33932185s" totalImagesPullingTime="18.811089721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:46.338862847 +0000 UTC m=+22.810441952" watchObservedRunningTime="2026-07-29 21:16:46.33932185 +0000 UTC m=+22.810900926"
Jul 29 21:16:47.049103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.049073    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:47.049266 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.049079    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:47.049266 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:47.049163    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:47.049266 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:47.049245    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:47.247697 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.247659    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="d5f4bc8a55fb1d05b07503f1ac9591e9a6dcce61b2e4f751101f75fb890ad6d4" exitCode=0
Jul 29 21:16:47.248596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.247739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"d5f4bc8a55fb1d05b07503f1ac9591e9a6dcce61b2e4f751101f75fb890ad6d4"}
Jul 29 21:16:47.248596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.248316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:47.249236 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.248823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:47.250584 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.250564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"db5191b2158230d175ee3479a3ecc73a86ea173bceaa11a8ccee0032276e2fea"}
Jul 29 21:16:47.250687 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.250616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"d8d7406f93c9a333f9fab3b3bf91f9b6982a11a768082129c61f5db9ec745121"}
Jul 29 21:16:47.250687 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.250655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"5d25b106f1b126043e5aac38f20d18f2a5d3d35a3ea3d683cade3d9729f3eaca"}
Jul 29 21:16:47.250687 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.250683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"f041025cd8cca7b43e4741e273280b6ede76f606c4974e0eeab0fd8dcb197d7a"}
Jul 29 21:16:47.251238 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.251218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:47.251549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.251533    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z9scg"
Jul 29 21:16:47.274626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.274589    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ljfcj" podStartSLOduration=4.576467419 podStartE2EDuration="23.274575801s" totalImagesPullingTime="18.698108382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:46.358272523 +0000 UTC m=+22.829851598" watchObservedRunningTime="2026-07-29 21:16:47.274575801 +0000 UTC m=+23.746154879"
Jul 29 21:16:47.433654 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:47.433625    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 29 21:16:48.019428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.019327    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T21:16:47.433645691Z","UUID":"3c5e34f9-893f-4f1f-a2a8-b7823cca2d76","Handler":null,"Name":"","Endpoint":""}
Jul 29 21:16:48.021643 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.021619    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 29 21:16:48.021769 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.021661    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 29 21:16:48.049435 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.049406    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:48.049621 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:48.049533    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:48.254105 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.254072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4hsfw" event={"ID":"e6233883-03f0-4ab6-b06d-d895a418213d","Type":"ContainerStarted","Data":"a4957218a098d2f75c4117c633bf17ca4fc8b722f6246b875aea2d415d4d8e81"}
Jul 29 21:16:48.255930 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.255909    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx" event={"ID":"f6622c9a-60c2-4e17-aaf6-f98042020807","Type":"ContainerStarted","Data":"8cf50dca23d48638241d4c5478e743604cb576064e135950299c201c1c50bff9"}
Jul 29 21:16:48.269700 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:48.269618    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4hsfw" podStartSLOduration=4.5675415 podStartE2EDuration="23.269605305s" totalImagesPullingTime="18.702063805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.269259237 +0000 UTC m=+24.740838316" watchObservedRunningTime="2026-07-29 21:16:48.269605305 +0000 UTC m=+24.741184380"
Jul 29 21:16:49.049235 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:49.048980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:49.049374 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:49.049349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:49.049457 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:49.049441    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:49.049543 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:49.049524    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:49.261189 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:49.261147    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx" event={"ID":"f6622c9a-60c2-4e17-aaf6-f98042020807","Type":"ContainerStarted","Data":"cf22bc83dbaba3aaefe9f684e0025ff1e23c0f7a023dca118936551ac31884f4"}
Jul 29 21:16:49.265103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:49.265070    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"314a893f12b882899cc2afb7907368006171b126af0d4772d9911351d70110f0"}
Jul 29 21:16:49.282093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:49.282016    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wrggx" podStartSLOduration=3.965175763 podStartE2EDuration="25.281996346s" totalImagesPullingTime="21.316820583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:49.281761509 +0000 UTC m=+25.753340587" watchObservedRunningTime="2026-07-29 21:16:49.281996346 +0000 UTC m=+25.753575423"
Jul 29 21:16:50.049250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:50.049214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:50.049479 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:50.049373    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:51.049428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:51.049387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:51.049428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:51.049420    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:51.049983 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:51.049530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:51.049983 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:51.049620    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:52.049253 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.049008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:52.049428 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:52.049384    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:52.276242 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.276215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" event={"ID":"826e07a4-4116-4f2e-8000-53324d87c284","Type":"ContainerStarted","Data":"5892e97399bd36e35711cd6685a40dae6451eddcd2a10aa46c760046345c64fb"}
Jul 29 21:16:52.276746 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.276445    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:52.276746 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.276503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:52.276746 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.276549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:52.290682 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.290660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:52.290934 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.290910    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:16:52.303378 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:52.303336    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf" podStartSLOduration=9.536164868 podStartE2EDuration="28.303322448s" totalImagesPullingTime="18.76715758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:52.302888667 +0000 UTC m=+28.774467799" watchObservedRunningTime="2026-07-29 21:16:52.303322448 +0000 UTC m=+28.774901524"
Jul 29 21:16:53.049090 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:53.049051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:53.049293 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:53.049153    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:53.049293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:53.049213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:53.049413 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:53.049318    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:53.280694 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:53.280656    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="d3632ed2d17541236a67c77f07586735eb6b30189d1d436bb50938ea78e66ef2" exitCode=0
Jul 29 21:16:53.281451 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:53.280741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"d3632ed2d17541236a67c77f07586735eb6b30189d1d436bb50938ea78e66ef2"}
Jul 29 21:16:54.049329 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.049299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:54.049505 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:54.049398    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:54.224808 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.224773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57rmr"]
Jul 29 21:16:54.224952 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.224898    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:54.225032 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:54.225007    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:54.226321 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.226287    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pm5pq"]
Jul 29 21:16:54.226462 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.226371    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:54.226536 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:54.226459    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:54.230783 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.230755    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cf85j"]
Jul 29 21:16:54.283343 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:54.283163    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:54.283710 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:54.283663    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:55.287003 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:55.286966    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="72f2dbcb03a99beed83daf8172c15046f0e97a7956933d23dcc3931b2be42183" exitCode=0
Jul 29 21:16:55.287457 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:55.287021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"72f2dbcb03a99beed83daf8172c15046f0e97a7956933d23dcc3931b2be42183"}
Jul 29 21:16:56.048963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:56.048930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:56.049117 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:56.048931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:56.049117 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:56.049060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:56.049209 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:56.049149    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:56.049209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:56.048938    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:56.049270 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:56.049242    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:57.293883 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:57.293843    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="50a9aedd180a694b7426ab2eea074a6fbb26e63174dba2c9b2910c03b1598c2d" exitCode=0
Jul 29 21:16:57.294577 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:57.293903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"50a9aedd180a694b7426ab2eea074a6fbb26e63174dba2c9b2910c03b1598c2d"}
Jul 29 21:16:57.702149 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:57.702098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:57.702337 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:57.702183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:57.702337 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702289    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 29 21:16:57.702337 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702312    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 29 21:16:57.702337 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702311    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:57.702337 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702322    2565 projected.go:196] Error preparing data for projected volume kube-api-access-h29rw for pod openshift-network-diagnostics/network-check-target-pm5pq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:57.702533 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702384    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs podName:76a8f833-1ec6-459f-a42f-23f8f6211ed6 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:29.702363251 +0000 UTC m=+66.173942321 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs") pod "network-metrics-daemon-cf85j" (UID: "76a8f833-1ec6-459f-a42f-23f8f6211ed6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 29 21:16:57.702533 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.702405    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw podName:3dc42795-9bbe-4e93-b10f-6107f3d17892 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:29.702395465 +0000 UTC m=+66.173974522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h29rw" (UniqueName: "kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw") pod "network-check-target-pm5pq" (UID: "3dc42795-9bbe-4e93-b10f-6107f3d17892") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 29 21:16:57.803212 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:57.803172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:57.803408 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.803316    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:57.803408 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:57.803376    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret podName:f78c6650-9271-48e4-b274-cfd7e4d542b3 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:29.803360881 +0000 UTC m=+66.274939940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret") pod "global-pull-secret-syncer-57rmr" (UID: "f78c6650-9271-48e4-b274-cfd7e4d542b3") : object "kube-system"/"original-pull-secret" not registered
Jul 29 21:16:58.048648 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:58.048613    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:16:58.048825 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:58.048760    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cf85j" podUID="76a8f833-1ec6-459f-a42f-23f8f6211ed6"
Jul 29 21:16:58.048825 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:58.048799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:16:58.048943 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:58.048872    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pm5pq" podUID="3dc42795-9bbe-4e93-b10f-6107f3d17892"
Jul 29 21:16:58.048943 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:58.048915    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:16:58.049067 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:16:58.049022    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-57rmr" podUID="f78c6650-9271-48e4-b274-cfd7e4d542b3"
Jul 29 21:16:59.838219 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.838139    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-123.ec2.internal" event="NodeReady"
Jul 29 21:16:59.838762 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.838285    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 29 21:16:59.873801 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.873765    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:16:59.879110 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.879084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:16:59.881587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.881560    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 29 21:16:59.881587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.881563    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 29 21:16:59.881777 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.881677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 29 21:16:59.881906 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.881885    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rgtl\""
Jul 29 21:16:59.884860 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.884836    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g2nvr"]
Jul 29 21:16:59.887947 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.887924    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 29 21:16:59.888801 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.888781    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:16:59.891716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.891505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 29 21:16:59.891716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.891655    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 29 21:16:59.892023 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.892002    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mw6qh"]
Jul 29 21:16:59.893677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.893653    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6msdl\""
Jul 29 21:16:59.898592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.898545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:16:59.898718 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.898636    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:16:59.900339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.900318    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g2nvr"]
Jul 29 21:16:59.900938 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.900917    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 29 21:16:59.901112 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.901092    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 29 21:16:59.901173 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.901160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 29 21:16:59.901299 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.901277    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-djnws\""
Jul 29 21:16:59.902483 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:16:59.902460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw6qh"]
Jul 29 21:17:00.020176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020205    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzn4v\" (UniqueName: \"kubernetes.io/projected/05830715-42d5-461f-942c-f14f4290a2a1-kube-api-access-dzn4v\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.020399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87058e58-2b75-43ed-9991-361cb06de8fe-cert\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.020399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020348    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9ks8\" (UniqueName: \"kubernetes.io/projected/87058e58-2b75-43ed-9991-361cb06de8fe-kube-api-access-m9ks8\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.020608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020407    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05830715-42d5-461f-942c-f14f4290a2a1-config-volume\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.020608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.020608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05830715-42d5-461f-942c-f14f4290a2a1-tmp-dir\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.020608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020593    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.020802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.020662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdqrx\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.048564 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.048526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:17:00.048740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.048522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:17:00.048740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.048526    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:17:00.051286 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 21:17:00.051509 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-929b2\""
Jul 29 21:17:00.051603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql2h5\""
Jul 29 21:17:00.051603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 21:17:00.051717 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051688    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 21:17:00.051768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.051756    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 21:17:00.121788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.121788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.121788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdqrx\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121832    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzn4v\" (UniqueName: \"kubernetes.io/projected/05830715-42d5-461f-942c-f14f4290a2a1-kube-api-access-dzn4v\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87058e58-2b75-43ed-9991-361cb06de8fe-cert\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.121977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9ks8\" (UniqueName: \"kubernetes.io/projected/87058e58-2b75-43ed-9991-361cb06de8fe-kube-api-access-m9ks8\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.122058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05830715-42d5-461f-942c-f14f4290a2a1-config-volume\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05830715-42d5-461f-942c-f14f4290a2a1-tmp-dir\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05830715-42d5-461f-942c-f14f4290a2a1-tmp-dir\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.122415    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.122425    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:00.122493 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.122472    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:00.622456176 +0000 UTC m=+37.094035229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:00.122827 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.122827 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.122646    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:00.122827 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.122691    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:00.622676281 +0000 UTC m=+37.094255337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:00.122827 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.123061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.122998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/05830715-42d5-461f-942c-f14f4290a2a1-config-volume\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.126436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.126297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.126436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.126325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.126593 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.126471    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87058e58-2b75-43ed-9991-361cb06de8fe-cert\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.131311 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.131285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzn4v\" (UniqueName: \"kubernetes.io/projected/05830715-42d5-461f-942c-f14f4290a2a1-kube-api-access-dzn4v\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.131463 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.131441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9ks8\" (UniqueName: \"kubernetes.io/projected/87058e58-2b75-43ed-9991-361cb06de8fe-kube-api-access-m9ks8\") pod \"ingress-canary-mw6qh\" (UID: \"87058e58-2b75-43ed-9991-361cb06de8fe\") " pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.131523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.131494    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.131816 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.131797    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdqrx\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.211092 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.211051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mw6qh"
Jul 29 21:17:00.624695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.624661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:00.624695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:00.624700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:00.624917 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.624819    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:00.624917 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.624841    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:00.624917 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.624868    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:00.624917 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.624877    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:01.624859896 +0000 UTC m=+38.096438973 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:00.625105 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:00.624925    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:01.624904666 +0000 UTC m=+38.096483733 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:01.227953 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:01.227707    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mw6qh"]
Jul 29 21:17:01.232272 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:01.232244    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87058e58_2b75_43ed_9991_361cb06de8fe.slice/crio-d07626cd3c7ebc675b3981d812fa0de342bc09ec305bbb61d5a6c289b42e6083 WatchSource:0}: Error finding container d07626cd3c7ebc675b3981d812fa0de342bc09ec305bbb61d5a6c289b42e6083: Status 404 returned error can't find the container with id d07626cd3c7ebc675b3981d812fa0de342bc09ec305bbb61d5a6c289b42e6083
Jul 29 21:17:01.308324 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:01.308280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerStarted","Data":"dec7dabbf6284f0fbbb7a03529b04adda4b87cd249cc7f9294c3c380f0b152ae"}
Jul 29 21:17:01.309395 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:01.309367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw6qh" event={"ID":"87058e58-2b75-43ed-9991-361cb06de8fe","Type":"ContainerStarted","Data":"d07626cd3c7ebc675b3981d812fa0de342bc09ec305bbb61d5a6c289b42e6083"}
Jul 29 21:17:01.633057 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:01.632958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:01.633057 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:01.633002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:01.633219 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:01.633129    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:01.633219 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:01.633185    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:03.633172179 +0000 UTC m=+40.104751233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:01.633219 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:01.633133    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:01.633313 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:01.633223    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:01.633313 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:01.633264    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:03.633253367 +0000 UTC m=+40.104832425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:02.315523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:02.315484    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="dec7dabbf6284f0fbbb7a03529b04adda4b87cd249cc7f9294c3c380f0b152ae" exitCode=0
Jul 29 21:17:02.315969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:02.315567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"dec7dabbf6284f0fbbb7a03529b04adda4b87cd249cc7f9294c3c380f0b152ae"}
Jul 29 21:17:03.321945 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:03.321740    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6243582-85a5-456a-8acb-a76632024c98" containerID="df16d7d3ec878d06219ddeefcfd648714c0a85f9778dd30f1d1fae37adad8753" exitCode=0
Jul 29 21:17:03.322376 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:03.321820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerDied","Data":"df16d7d3ec878d06219ddeefcfd648714c0a85f9778dd30f1d1fae37adad8753"}
Jul 29 21:17:03.651345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:03.651257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:03.651490 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:03.651363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:03.651490 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:03.651421    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:03.651490 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:03.651454    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:03.651490 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:03.651466    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:03.651490 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:03.651490    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.651474099 +0000 UTC m=+44.123053153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:03.651673 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:03.651505    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.651499322 +0000 UTC m=+44.123078376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:04.326180 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:04.326142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mw6qh" event={"ID":"87058e58-2b75-43ed-9991-361cb06de8fe","Type":"ContainerStarted","Data":"7d9baaee323f6c6255da01a7ac7273cdca95a4f6a7287767289f99afcfc1d5f9"}
Jul 29 21:17:04.329304 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:04.329282    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fz857" event={"ID":"a6243582-85a5-456a-8acb-a76632024c98","Type":"ContainerStarted","Data":"0f98fc2feffba409d9c3d307942e2d575632c38a53413f3058d32e5e9baac10a"}
Jul 29 21:17:04.342143 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:04.342092    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mw6qh" podStartSLOduration=2.7888608870000002 podStartE2EDuration="5.34207404s" totalImagesPullingTime="2.553213153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:04.341869195 +0000 UTC m=+40.813448271" watchObservedRunningTime="2026-07-29 21:17:04.34207404 +0000 UTC m=+40.813653117"
Jul 29 21:17:04.367037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:04.366977    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fz857" podStartSLOduration=11.195968372 podStartE2EDuration="40.366960448s" totalImagesPullingTime="29.170992076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:04.366300198 +0000 UTC m=+40.837879307" watchObservedRunningTime="2026-07-29 21:17:04.366960448 +0000 UTC m=+40.838539521"
Jul 29 21:17:06.984936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:06.984901    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"]
Jul 29 21:17:07.016539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.016511    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"]
Jul 29 21:17:07.016662 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.016600    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"
Jul 29 21:17:07.019899 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.019873    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:17:07.020057 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.019924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xvqb7\""
Jul 29 21:17:07.020779 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.020763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 29 21:17:07.176100 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.176064    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdmn\" (UniqueName: \"kubernetes.io/projected/d5ea33ae-e191-4d92-ba22-651b3729928f-kube-api-access-6fdmn\") pod \"volume-data-source-validator-549c6d54cf-8g6g6\" (UID: \"d5ea33ae-e191-4d92-ba22-651b3729928f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"
Jul 29 21:17:07.276963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.276897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdmn\" (UniqueName: \"kubernetes.io/projected/d5ea33ae-e191-4d92-ba22-651b3729928f-kube-api-access-6fdmn\") pod \"volume-data-source-validator-549c6d54cf-8g6g6\" (UID: \"d5ea33ae-e191-4d92-ba22-651b3729928f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"
Jul 29 21:17:07.289675 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.289654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdmn\" (UniqueName: \"kubernetes.io/projected/d5ea33ae-e191-4d92-ba22-651b3729928f-kube-api-access-6fdmn\") pod \"volume-data-source-validator-549c6d54cf-8g6g6\" (UID: \"d5ea33ae-e191-4d92-ba22-651b3729928f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"
Jul 29 21:17:07.325412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.325381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"
Jul 29 21:17:07.465515 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.465487    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6"]
Jul 29 21:17:07.468702 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:07.468673    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ea33ae_e191_4d92_ba22_651b3729928f.slice/crio-7a7d55fdb73e4038696583575406fa6aa730925d97376e687bc6f7b064ef75ee WatchSource:0}: Error finding container 7a7d55fdb73e4038696583575406fa6aa730925d97376e687bc6f7b064ef75ee: Status 404 returned error can't find the container with id 7a7d55fdb73e4038696583575406fa6aa730925d97376e687bc6f7b064ef75ee
Jul 29 21:17:07.679726 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.679696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:07.679884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:07.679801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:07.679884 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:07.679849    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:07.679955 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:07.679893    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:07.679955 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:07.679907    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:07.679955 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:07.679920    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:15.679904453 +0000 UTC m=+52.151483506 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:07.679955 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:07.679950    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:15.67993742 +0000 UTC m=+52.151516479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:08.339720 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:08.339685    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6" event={"ID":"d5ea33ae-e191-4d92-ba22-651b3729928f","Type":"ContainerStarted","Data":"7a7d55fdb73e4038696583575406fa6aa730925d97376e687bc6f7b064ef75ee"}
Jul 29 21:17:10.345743 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:10.345656    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6" event={"ID":"d5ea33ae-e191-4d92-ba22-651b3729928f","Type":"ContainerStarted","Data":"e1bc2d99389224bda8c0ab5dce56490124a7387a752b8a55518a905e2475e4c8"}
Jul 29 21:17:10.382891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:10.382840    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8g6g6" podStartSLOduration=1.9248647669999999 podStartE2EDuration="4.382824734s" totalImagesPullingTime="2.457959967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:10.382500402 +0000 UTC m=+46.854079480" watchObservedRunningTime="2026-07-29 21:17:10.382824734 +0000 UTC m=+46.854403807"
Jul 29 21:17:12.800550 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.800511    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cvf9r"]
Jul 29 21:17:12.832804 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.832769    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvf9r"]
Jul 29 21:17:12.832958 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.832893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:12.835424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.835398    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 29 21:17:12.835424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.835418    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 29 21:17:12.835621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.835525    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 29 21:17:12.836277 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.836257    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 29 21:17:12.836379 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.836275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4rkn\""
Jul 29 21:17:12.919209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.919171    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:12.919385 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.919270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnr8f\" (UniqueName: \"kubernetes.io/projected/aba3f655-06ad-4d05-8676-230b17e4c922-kube-api-access-wnr8f\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:12.919385 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.919331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aba3f655-06ad-4d05-8676-230b17e4c922-crio-socket\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:12.919385 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.919355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aba3f655-06ad-4d05-8676-230b17e4c922-data-volume\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:12.919385 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:12.919372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.019723 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.019686    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.019841 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.019783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnr8f\" (UniqueName: \"kubernetes.io/projected/aba3f655-06ad-4d05-8676-230b17e4c922-kube-api-access-wnr8f\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.019882 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.019840    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:13.019882 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.019851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aba3f655-06ad-4d05-8676-230b17e4c922-crio-socket\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.019963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.019882    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aba3f655-06ad-4d05-8676-230b17e4c922-data-volume\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.019963 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.019904    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:13.519885324 +0000 UTC m=+49.991464398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:13.019963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.019932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.020108 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.020006    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:17:13.020108 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.020075    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:13.52003846 +0000 UTC m=+49.991617537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : configmap "kube-rbac-proxy" not found
Jul 29 21:17:13.020178 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.020095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aba3f655-06ad-4d05-8676-230b17e4c922-crio-socket\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.020212 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.020201    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aba3f655-06ad-4d05-8676-230b17e4c922-data-volume\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.029668 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.029651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnr8f\" (UniqueName: \"kubernetes.io/projected/aba3f655-06ad-4d05-8676-230b17e4c922-kube-api-access-wnr8f\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.514270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.514232    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"]
Jul 29 21:17:13.523679 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.523650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.523759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.523752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:13.523824 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.523804    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:13.523889 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.523880    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.523865375 +0000 UTC m=+50.995444429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:13.523939 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.523816    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:17:13.523973 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:13.523939    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.523927862 +0000 UTC m=+50.995506918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : configmap "kube-rbac-proxy" not found
Jul 29 21:17:13.530240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.530205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"]
Jul 29 21:17:13.530395 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.530319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"
Jul 29 21:17:13.532675 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.532650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 29 21:17:13.532786 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.532650    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-pw7vs\""
Jul 29 21:17:13.533556 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.533540    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 29 21:17:13.624825 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.624787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fb8q\" (UniqueName: \"kubernetes.io/projected/7d1b8018-59b9-4a8e-a718-4b7feec19c36-kube-api-access-4fb8q\") pod \"migrator-6865c9967-wcx7v\" (UID: \"7d1b8018-59b9-4a8e-a718-4b7feec19c36\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"
Jul 29 21:17:13.725834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.725793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fb8q\" (UniqueName: \"kubernetes.io/projected/7d1b8018-59b9-4a8e-a718-4b7feec19c36-kube-api-access-4fb8q\") pod \"migrator-6865c9967-wcx7v\" (UID: \"7d1b8018-59b9-4a8e-a718-4b7feec19c36\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"
Jul 29 21:17:13.734446 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.734419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fb8q\" (UniqueName: \"kubernetes.io/projected/7d1b8018-59b9-4a8e-a718-4b7feec19c36-kube-api-access-4fb8q\") pod \"migrator-6865c9967-wcx7v\" (UID: \"7d1b8018-59b9-4a8e-a718-4b7feec19c36\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"
Jul 29 21:17:13.838501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.838423    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"
Jul 29 21:17:13.992372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:13.992338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v"]
Jul 29 21:17:13.995364 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:13.995336    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1b8018_59b9_4a8e_a718_4b7feec19c36.slice/crio-fbf1367b2f6bc93cce12c611ef63eb094a8bcca2b5b9852eaa272ac33adab9c8 WatchSource:0}: Error finding container fbf1367b2f6bc93cce12c611ef63eb094a8bcca2b5b9852eaa272ac33adab9c8: Status 404 returned error can't find the container with id fbf1367b2f6bc93cce12c611ef63eb094a8bcca2b5b9852eaa272ac33adab9c8
Jul 29 21:17:14.356188 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:14.356154    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v" event={"ID":"7d1b8018-59b9-4a8e-a718-4b7feec19c36","Type":"ContainerStarted","Data":"fbf1367b2f6bc93cce12c611ef63eb094a8bcca2b5b9852eaa272ac33adab9c8"}
Jul 29 21:17:14.533124 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:14.533086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:14.533124 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:14.533131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:14.533326 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:14.533213    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 29 21:17:14.533326 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:14.533253    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:14.533326 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:14.533276    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:16.533261129 +0000 UTC m=+53.004840183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : configmap "kube-rbac-proxy" not found
Jul 29 21:17:14.533326 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:14.533292    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:16.533281983 +0000 UTC m=+53.004861041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:14.993641 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:14.993616    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgrr6_8f9ce67f-8d99-4397-8fe6-395e53f23440/dns-node-resolver/0.log"
Jul 29 21:17:15.395396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:15.395317    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ljfcj_c1a702f0-7382-4c43-b759-1466980f155e/node-ca/0.log"
Jul 29 21:17:15.741532 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:15.741485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:15.741532 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:15.741537    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:15.741731 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:15.741634    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 29 21:17:15.741731 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:15.741654    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b854cf867-9jd8t: secret "image-registry-tls" not found
Jul 29 21:17:15.741731 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:15.741692    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 29 21:17:15.741731 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:15.741707    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls podName:25296465-f81c-4654-8bdf-6f6edb8e1418 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:31.741692692 +0000 UTC m=+68.213271747 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls") pod "image-registry-5b854cf867-9jd8t" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418") : secret "image-registry-tls" not found
Jul 29 21:17:15.741857 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:15.741748    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls podName:05830715-42d5-461f-942c-f14f4290a2a1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:31.741735577 +0000 UTC m=+68.213314632 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls") pod "dns-default-g2nvr" (UID: "05830715-42d5-461f-942c-f14f4290a2a1") : secret "dns-default-metrics-tls" not found
Jul 29 21:17:16.196510 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.196475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mw6qh_87058e58-2b75-43ed-9991-361cb06de8fe/serve-healthcheck-canary/0.log"
Jul 29 21:17:16.362796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.362709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v" event={"ID":"7d1b8018-59b9-4a8e-a718-4b7feec19c36","Type":"ContainerStarted","Data":"94bafa4e64431e55ba64c4c999029fa1a60170a5e0df8e55ac7c40b84411f687"}
Jul 29 21:17:16.362796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.362767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v" event={"ID":"7d1b8018-59b9-4a8e-a718-4b7feec19c36","Type":"ContainerStarted","Data":"c9c850f862d6163fcb2bfe85c92fdc88b30657e936fffe590f2ee5693ed37635"}
Jul 29 21:17:16.384833 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.384780    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-wcx7v" podStartSLOduration=1.554575037 podStartE2EDuration="3.384765856s" totalImagesPullingTime="1.830190819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:16.384500168 +0000 UTC m=+52.856079244" watchObservedRunningTime="2026-07-29 21:17:16.384765856 +0000 UTC m=+52.856344911"
Jul 29 21:17:16.547392 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.547356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:16.547545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.547403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:16.547545 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:16.547494    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:16.547545 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:16.547543    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:20.547530208 +0000 UTC m=+57.019109263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:16.547910 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:16.547887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aba3f655-06ad-4d05-8676-230b17e4c922-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:20.577970 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:20.577933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:20.578442 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:20.578109    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:20.578442 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:17:20.578180    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls podName:aba3f655-06ad-4d05-8676-230b17e4c922 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:28.578162498 +0000 UTC m=+65.049741566 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cvf9r" (UID: "aba3f655-06ad-4d05-8676-230b17e4c922") : secret "insights-runtime-extractor-tls" not found
Jul 29 21:17:24.297941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:24.297913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ktwf"
Jul 29 21:17:28.641432 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:28.641365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:28.643629 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:28.643608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aba3f655-06ad-4d05-8676-230b17e4c922-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cvf9r\" (UID: \"aba3f655-06ad-4d05-8676-230b17e4c922\") " pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:28.744412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:28.744375    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4rkn\""
Jul 29 21:17:28.752176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:28.752152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cvf9r"
Jul 29 21:17:28.873466 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:28.873434    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cvf9r"]
Jul 29 21:17:28.877640 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:28.877605    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba3f655_06ad_4d05_8676_230b17e4c922.slice/crio-fcf6fe15e6d2d0cc288ff30a03fe72d0e4f10bdad1e71fb6b1dc8ca4e2ef43fa WatchSource:0}: Error finding container fcf6fe15e6d2d0cc288ff30a03fe72d0e4f10bdad1e71fb6b1dc8ca4e2ef43fa: Status 404 returned error can't find the container with id fcf6fe15e6d2d0cc288ff30a03fe72d0e4f10bdad1e71fb6b1dc8ca4e2ef43fa
Jul 29 21:17:29.395613 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.395579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvf9r" event={"ID":"aba3f655-06ad-4d05-8676-230b17e4c922","Type":"ContainerStarted","Data":"fcf6fe15e6d2d0cc288ff30a03fe72d0e4f10bdad1e71fb6b1dc8ca4e2ef43fa"}
Jul 29 21:17:29.750849 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.750802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:17:29.751324 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.750905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:17:29.753257 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.753232    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 29 21:17:29.754065 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.754030    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 29 21:17:29.764622 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.764439    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 29 21:17:29.765804 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.765780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76a8f833-1ec6-459f-a42f-23f8f6211ed6-metrics-certs\") pod \"network-metrics-daemon-cf85j\" (UID: \"76a8f833-1ec6-459f-a42f-23f8f6211ed6\") " pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:17:29.775537 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.775488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29rw\" (UniqueName: \"kubernetes.io/projected/3dc42795-9bbe-4e93-b10f-6107f3d17892-kube-api-access-h29rw\") pod \"network-check-target-pm5pq\" (UID: \"3dc42795-9bbe-4e93-b10f-6107f3d17892\") " pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:17:29.851674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.851599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:17:29.854280 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.854252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 29 21:17:29.865023 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:29.864994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f78c6650-9271-48e4-b274-cfd7e4d542b3-original-pull-secret\") pod \"global-pull-secret-syncer-57rmr\" (UID: \"f78c6650-9271-48e4-b274-cfd7e4d542b3\") " pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:17:30.062575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.062493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql2h5\""
Jul 29 21:17:30.067510 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.067480    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-57rmr"
Jul 29 21:17:30.070272 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.070162    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cf85j"
Jul 29 21:17:30.076553 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.076313    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-929b2\""
Jul 29 21:17:30.084476 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.084448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:17:30.231106 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.231015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-57rmr"]
Jul 29 21:17:30.236988 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:30.236951    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78c6650_9271_48e4_b274_cfd7e4d542b3.slice/crio-0e76fc2861c897ea047615e41307b464c43bf1fde8281230c2879a7291725646 WatchSource:0}: Error finding container 0e76fc2861c897ea047615e41307b464c43bf1fde8281230c2879a7291725646: Status 404 returned error can't find the container with id 0e76fc2861c897ea047615e41307b464c43bf1fde8281230c2879a7291725646
Jul 29 21:17:30.253000 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.252922    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cf85j"]
Jul 29 21:17:30.258287 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:30.258260    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a8f833_1ec6_459f_a42f_23f8f6211ed6.slice/crio-e0042e03dae0ce679892530486cedc0d151f7e4261d877efd0be456ae62c492b WatchSource:0}: Error finding container e0042e03dae0ce679892530486cedc0d151f7e4261d877efd0be456ae62c492b: Status 404 returned error can't find the container with id e0042e03dae0ce679892530486cedc0d151f7e4261d877efd0be456ae62c492b
Jul 29 21:17:30.276376 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.276349    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pm5pq"]
Jul 29 21:17:30.280011 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:30.279986    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc42795_9bbe_4e93_b10f_6107f3d17892.slice/crio-f18c57845160e8525c66e80dd52ad4617bc874c023ba4ce9d2eabdc1e4248bd6 WatchSource:0}: Error finding container f18c57845160e8525c66e80dd52ad4617bc874c023ba4ce9d2eabdc1e4248bd6: Status 404 returned error can't find the container with id f18c57845160e8525c66e80dd52ad4617bc874c023ba4ce9d2eabdc1e4248bd6
Jul 29 21:17:30.400219 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.400128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57rmr" event={"ID":"f78c6650-9271-48e4-b274-cfd7e4d542b3","Type":"ContainerStarted","Data":"0e76fc2861c897ea047615e41307b464c43bf1fde8281230c2879a7291725646"}
Jul 29 21:17:30.401524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.401479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf85j" event={"ID":"76a8f833-1ec6-459f-a42f-23f8f6211ed6","Type":"ContainerStarted","Data":"e0042e03dae0ce679892530486cedc0d151f7e4261d877efd0be456ae62c492b"}
Jul 29 21:17:30.402679 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:30.402641    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pm5pq" event={"ID":"3dc42795-9bbe-4e93-b10f-6107f3d17892","Type":"ContainerStarted","Data":"f18c57845160e8525c66e80dd52ad4617bc874c023ba4ce9d2eabdc1e4248bd6"}
Jul 29 21:17:31.770141 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:31.770075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:31.770141 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:31.770122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:31.772932 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:31.772876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/05830715-42d5-461f-942c-f14f4290a2a1-metrics-tls\") pod \"dns-default-g2nvr\" (UID: \"05830715-42d5-461f-942c-f14f4290a2a1\") " pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:31.773104 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:31.773084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"image-registry-5b854cf867-9jd8t\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") " pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:31.994766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:31.994728    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rgtl\""
Jul 29 21:17:32.002780 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:32.002747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:32.007066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:32.006942    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6msdl\""
Jul 29 21:17:32.014568 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:32.014539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:37.118206 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:37.118159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g2nvr"]
Jul 29 21:17:37.341700 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:37.341662    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05830715_42d5_461f_942c_f14f4290a2a1.slice/crio-8b36ab9ffff3b7ae15e3752903c676df51f28a714c777ee0f038340fcd8e0ea5 WatchSource:0}: Error finding container 8b36ab9ffff3b7ae15e3752903c676df51f28a714c777ee0f038340fcd8e0ea5: Status 404 returned error can't find the container with id 8b36ab9ffff3b7ae15e3752903c676df51f28a714c777ee0f038340fcd8e0ea5
Jul 29 21:17:37.429122 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:37.429059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g2nvr" event={"ID":"05830715-42d5-461f-942c-f14f4290a2a1","Type":"ContainerStarted","Data":"8b36ab9ffff3b7ae15e3752903c676df51f28a714c777ee0f038340fcd8e0ea5"}
Jul 29 21:17:37.503141 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:37.503069    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:17:37.507424 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:37.507387    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25296465_f81c_4654_8bdf_6f6edb8e1418.slice/crio-a39352e0b4a08182c8b92ebc93d9b0de4bf515e7ea057f8719e970fef22f685c WatchSource:0}: Error finding container a39352e0b4a08182c8b92ebc93d9b0de4bf515e7ea057f8719e970fef22f685c: Status 404 returned error can't find the container with id a39352e0b4a08182c8b92ebc93d9b0de4bf515e7ea057f8719e970fef22f685c
Jul 29 21:17:38.434849 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.434810    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvf9r" event={"ID":"aba3f655-06ad-4d05-8676-230b17e4c922","Type":"ContainerStarted","Data":"ca5721fad8242eb9c72a2c4275fa5be565df5602cfe133dd01ab431409c7dba8"}
Jul 29 21:17:38.436442 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.436411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-57rmr" event={"ID":"f78c6650-9271-48e4-b274-cfd7e4d542b3","Type":"ContainerStarted","Data":"5d6fb03f1f27262c36c3b90cb92cf1aae9e45002bae5f695535e2e0e8f4bdd61"}
Jul 29 21:17:38.438310 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.438281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf85j" event={"ID":"76a8f833-1ec6-459f-a42f-23f8f6211ed6","Type":"ContainerStarted","Data":"64a5bd6a636e816439307f6c59f82e3660b388ddaf0f081b85b2fc03ba6c22aa"}
Jul 29 21:17:38.438435 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.438334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cf85j" event={"ID":"76a8f833-1ec6-459f-a42f-23f8f6211ed6","Type":"ContainerStarted","Data":"d293fbb3f38fb8a590304cc3e0f4916c1aaf3821583fb42419df336ea9fa052a"}
Jul 29 21:17:38.440108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.440079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pm5pq" event={"ID":"3dc42795-9bbe-4e93-b10f-6107f3d17892","Type":"ContainerStarted","Data":"0dd5b181795acf6adaf3f7974304358b12743dd452cb013dc1b2245b39416cdc"}
Jul 29 21:17:38.440227 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.440133    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:17:38.441618 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.441595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" event={"ID":"25296465-f81c-4654-8bdf-6f6edb8e1418","Type":"ContainerStarted","Data":"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"}
Jul 29 21:17:38.441711 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.441637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" event={"ID":"25296465-f81c-4654-8bdf-6f6edb8e1418","Type":"ContainerStarted","Data":"a39352e0b4a08182c8b92ebc93d9b0de4bf515e7ea057f8719e970fef22f685c"}
Jul 29 21:17:38.441711 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.441680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:38.465475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.465415    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-57rmr" podStartSLOduration=66.361616789 podStartE2EDuration="1m13.465388321s" totalImagesPullingTime="7.103771532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:38.464452437 +0000 UTC m=+74.936031513" watchObservedRunningTime="2026-07-29 21:17:38.465388321 +0000 UTC m=+74.936967397"
Jul 29 21:17:38.503942 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.503879    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" podStartSLOduration=73.503858051 podStartE2EDuration="1m13.503858051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:38.501995517 +0000 UTC m=+74.973574622" watchObservedRunningTime="2026-07-29 21:17:38.503858051 +0000 UTC m=+74.975437129"
Jul 29 21:17:38.523848 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.523788    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cf85j" podStartSLOduration=67.445028463 podStartE2EDuration="1m14.523766158s" totalImagesPullingTime="7.078737695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:38.523018485 +0000 UTC m=+74.994597564" watchObservedRunningTime="2026-07-29 21:17:38.523766158 +0000 UTC m=+74.995345235"
Jul 29 21:17:38.557697 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.557610    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pm5pq" podStartSLOduration=67.497113616 podStartE2EDuration="1m14.557586231s" totalImagesPullingTime="7.060472615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:38.55712053 +0000 UTC m=+75.028699629" watchObservedRunningTime="2026-07-29 21:17:38.557586231 +0000 UTC m=+75.029165301"
Jul 29 21:17:38.583701 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.582386    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"]
Jul 29 21:17:38.586810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.586237    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.599241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.598310    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 29 21:17:38.599241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.598607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 29 21:17:38.599241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.599026    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 29 21:17:38.600118 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.599870    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 29 21:17:38.611619 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.611541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"]
Jul 29 21:17:38.679493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.679088    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-6wc8p"]
Jul 29 21:17:38.682936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.682903    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:17:38.685993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.685517    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gmxwj\""
Jul 29 21:17:38.685993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.685806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 29 21:17:38.685993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.685940    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 29 21:17:38.716353 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.716314    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6wc8p"]
Jul 29 21:17:38.727898 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.727868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e16fba1-371e-43b3-8f55-ae17a679e125-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.728038 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.727916    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e16fba1-371e-43b3-8f55-ae17a679e125-tmp\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.728038 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.727943    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlnmz\" (UniqueName: \"kubernetes.io/projected/0e16fba1-371e-43b3-8f55-ae17a679e125-kube-api-access-dlnmz\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.829201 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.828634    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e16fba1-371e-43b3-8f55-ae17a679e125-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.829201 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.828693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnhct\" (UniqueName: \"kubernetes.io/projected/d83b763d-5d3c-42c7-bd05-b7b3992a1e81-kube-api-access-hnhct\") pod \"downloads-7c97dc4555-6wc8p\" (UID: \"d83b763d-5d3c-42c7-bd05-b7b3992a1e81\") " pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:17:38.829201 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.828727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e16fba1-371e-43b3-8f55-ae17a679e125-tmp\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.829201 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.828792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlnmz\" (UniqueName: \"kubernetes.io/projected/0e16fba1-371e-43b3-8f55-ae17a679e125-kube-api-access-dlnmz\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.829546 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.829424    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e16fba1-371e-43b3-8f55-ae17a679e125-tmp\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.832162 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.832113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0e16fba1-371e-43b3-8f55-ae17a679e125-klusterlet-config\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.849224 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.849190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlnmz\" (UniqueName: \"kubernetes.io/projected/0e16fba1-371e-43b3-8f55-ae17a679e125-kube-api-access-dlnmz\") pod \"klusterlet-addon-workmgr-6864cc47b7-dxwt6\" (UID: \"0e16fba1-371e-43b3-8f55-ae17a679e125\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.903570 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.903330    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:38.929910 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.929604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnhct\" (UniqueName: \"kubernetes.io/projected/d83b763d-5d3c-42c7-bd05-b7b3992a1e81-kube-api-access-hnhct\") pod \"downloads-7c97dc4555-6wc8p\" (UID: \"d83b763d-5d3c-42c7-bd05-b7b3992a1e81\") " pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:17:38.947575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.947473    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnhct\" (UniqueName: \"kubernetes.io/projected/d83b763d-5d3c-42c7-bd05-b7b3992a1e81-kube-api-access-hnhct\") pod \"downloads-7c97dc4555-6wc8p\" (UID: \"d83b763d-5d3c-42c7-bd05-b7b3992a1e81\") " pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:17:38.995632 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:38.995386    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:17:39.107057 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:39.106907    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"]
Jul 29 21:17:39.110949 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:39.110883    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e16fba1_371e_43b3_8f55_ae17a679e125.slice/crio-47ed54c73ca863c954bee9bbb7c6a1517d7ccb1dae705267a9725d70c284c466 WatchSource:0}: Error finding container 47ed54c73ca863c954bee9bbb7c6a1517d7ccb1dae705267a9725d70c284c466: Status 404 returned error can't find the container with id 47ed54c73ca863c954bee9bbb7c6a1517d7ccb1dae705267a9725d70c284c466
Jul 29 21:17:39.179478 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:39.179443    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-6wc8p"]
Jul 29 21:17:39.448500 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:39.448462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6" event={"ID":"0e16fba1-371e-43b3-8f55-ae17a679e125","Type":"ContainerStarted","Data":"47ed54c73ca863c954bee9bbb7c6a1517d7ccb1dae705267a9725d70c284c466"}
Jul 29 21:17:39.693718 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:39.693679    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83b763d_5d3c_42c7_bd05_b7b3992a1e81.slice/crio-76c5df9c160f221b86b4087241b0c692eb332898d83ff445c14bf3d346035476 WatchSource:0}: Error finding container 76c5df9c160f221b86b4087241b0c692eb332898d83ff445c14bf3d346035476: Status 404 returned error can't find the container with id 76c5df9c160f221b86b4087241b0c692eb332898d83ff445c14bf3d346035476
Jul 29 21:17:40.454908 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:40.454869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6wc8p" event={"ID":"d83b763d-5d3c-42c7-bd05-b7b3992a1e81","Type":"ContainerStarted","Data":"76c5df9c160f221b86b4087241b0c692eb332898d83ff445c14bf3d346035476"}
Jul 29 21:17:41.461963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:41.461922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g2nvr" event={"ID":"05830715-42d5-461f-942c-f14f4290a2a1","Type":"ContainerStarted","Data":"deeb117eafd5d513822625c8ccfe0933d5fc62e8247dd00cea3a22c5200389ae"}
Jul 29 21:17:42.479515 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:42.479474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g2nvr" event={"ID":"05830715-42d5-461f-942c-f14f4290a2a1","Type":"ContainerStarted","Data":"1b2f95f6eff4a093eeb0b0244670ecaddef4482eb5933b04d8c925f65ecb7002"}
Jul 29 21:17:42.479993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:42.479748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:43.485549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:43.485472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvf9r" event={"ID":"aba3f655-06ad-4d05-8676-230b17e4c922","Type":"ContainerStarted","Data":"58f2f1346ea92e179885bfbecced67daac65fc43bdb4ff13398e55a34412869b"}
Jul 29 21:17:45.495492 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:45.495452    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6" event={"ID":"0e16fba1-371e-43b3-8f55-ae17a679e125","Type":"ContainerStarted","Data":"b2fa29d5e2b3173503e25815c4dd18e61b6077cc7334540bea1dbc40c7904d53"}
Jul 29 21:17:45.495939 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:45.495678    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:45.497543 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:45.497502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6"
Jul 29 21:17:45.516206 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:45.516152    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g2nvr" podStartSLOduration=44.110095776 podStartE2EDuration="46.51613077s" totalImagesPullingTime="2.406034994s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:42.499165308 +0000 UTC m=+78.970744385" watchObservedRunningTime="2026-07-29 21:17:45.51613077 +0000 UTC m=+81.987709849"
Jul 29 21:17:45.516472 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:45.516441    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6864cc47b7-dxwt6" podStartSLOduration=1.761718971 podStartE2EDuration="7.516433459s" totalImagesPullingTime="5.754714488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:45.51530613 +0000 UTC m=+81.986885203" watchObservedRunningTime="2026-07-29 21:17:45.516433459 +0000 UTC m=+81.988012538"
Jul 29 21:17:48.231169 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.231057    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:17:48.242173 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.242144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.245522 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245497    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qrndx\""
Jul 29 21:17:48.245663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 21:17:48.245663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245583    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 21:17:48.245663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 21:17:48.245663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245609    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 21:17:48.245663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.245536    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 21:17:48.248833 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.248807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:17:48.402093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96cbh\" (UniqueName: \"kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.402267 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.402267 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402182    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.402267 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.402414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.402414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.402307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503544 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503406    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503544 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503544 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96cbh\" (UniqueName: \"kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.503701 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.504086 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.503999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.504299 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.504252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.504893 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.504827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.506818 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.506762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.508066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.507994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.518718 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.518680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96cbh\" (UniqueName: \"kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh\") pod \"console-ccf7494b4-pvqtg\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") " pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:48.555505 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:48.555471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:17:49.664128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:49.664099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:17:49.666942 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:49.666919    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94381045_f26f_415a_bca1_48b4c6127924.slice/crio-7e8bc74d6d37b925ce5a5e9be98537fc2b5b892e6e1c4da00f3e19652edd24e8 WatchSource:0}: Error finding container 7e8bc74d6d37b925ce5a5e9be98537fc2b5b892e6e1c4da00f3e19652edd24e8: Status 404 returned error can't find the container with id 7e8bc74d6d37b925ce5a5e9be98537fc2b5b892e6e1c4da00f3e19652edd24e8
Jul 29 21:17:50.520035 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:50.519992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cvf9r" event={"ID":"aba3f655-06ad-4d05-8676-230b17e4c922","Type":"ContainerStarted","Data":"de358454d7a5093384c26d3de8e406e8601f74fc0b8706966e98f9de130941ae"}
Jul 29 21:17:50.521787 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:50.521753    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccf7494b4-pvqtg" event={"ID":"94381045-f26f-415a-bca1-48b4c6127924","Type":"ContainerStarted","Data":"7e8bc74d6d37b925ce5a5e9be98537fc2b5b892e6e1c4da00f3e19652edd24e8"}
Jul 29 21:17:52.008143 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:52.008099    2565 patch_prober.go:28] interesting pod/image-registry-5b854cf867-9jd8t container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 29 21:17:52.008556 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:52.008170    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:17:52.487975 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:52.487945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g2nvr"
Jul 29 21:17:52.511437 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:52.511368    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cvf9r" podStartSLOduration=21.083280277 podStartE2EDuration="40.511345908s" totalImagesPullingTime="19.428065631s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:17:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:50.53730898 +0000 UTC m=+87.008888070" watchObservedRunningTime="2026-07-29 21:17:52.511345908 +0000 UTC m=+88.982924999"
Jul 29 21:17:54.428759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.428723    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:17:54.541938 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.541896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:17:54.542138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.542038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.557896 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.557870    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 21:17:54.654912 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.654868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.654925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.654951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgpx7\" (UniqueName: \"kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.655058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.655123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.655166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.655270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.655197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756252 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756252 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756252 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.756514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756385    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgpx7\" (UniqueName: \"kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.757289 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756989    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.757289 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.756988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.757639 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.757619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.757828 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.757798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.759339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.759318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.759580 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.759560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.764991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.764967    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgpx7\" (UniqueName: \"kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7\") pod \"console-7fb9cd67f7-mbvfs\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") " pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:54.853250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:54.853211    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:17:56.898101 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.898036    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"]
Jul 29 21:17:56.948278 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.948215    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-84wmf"]
Jul 29 21:17:56.948456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.948284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:56.953914 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.953887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 29 21:17:56.954086 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.953933    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 29 21:17:56.954166 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.954112    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 29 21:17:56.955581 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.955559    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 29 21:17:56.956312 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.956291    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 29 21:17:56.956648 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.956626    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b2nqt\""
Jul 29 21:17:56.956743 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.956689    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 29 21:17:56.972587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.972554    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"]
Jul 29 21:17:56.972767 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.972690    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:56.976251 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.976225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wblq4\""
Jul 29 21:17:56.976736 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.976711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 29 21:17:56.976846 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.976772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 29 21:17:56.976846 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:56.976785    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 29 21:17:57.073953 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.073923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-sys\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.073967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84b77d03-d1fe-423d-91c1-3861873b302a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.074145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.073993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtjtw\" (UniqueName: \"kubernetes.io/projected/84b77d03-d1fe-423d-91c1-3861873b302a-kube-api-access-jtjtw\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.074145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8n92\" (UniqueName: \"kubernetes.io/projected/27f3633c-8a0c-421f-9693-b3ee90c448b9-kube-api-access-b8n92\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-textfile\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.074274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.074274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-root\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-tls\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-metrics-client-ca\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-wtmp\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.074473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.074442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.174856 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.174819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-tls\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.174856 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.174868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-metrics-client-ca\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.174895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-wtmp\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-wtmp\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-sys\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84b77d03-d1fe-423d-91c1-3861873b302a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtjtw\" (UniqueName: \"kubernetes.io/projected/84b77d03-d1fe-423d-91c1-3861873b302a-kube-api-access-jtjtw\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8n92\" (UniqueName: \"kubernetes.io/projected/27f3633c-8a0c-421f-9693-b3ee90c448b9-kube-api-access-b8n92\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-textfile\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-root\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175574    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-metrics-client-ca\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.175729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/84b77d03-d1fe-423d-91c1-3861873b302a-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.175729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.175721    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-sys\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.176036 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.176012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-accelerators-collector-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.176127 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.176103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.176127 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.176118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-textfile\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.176239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.176172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27f3633c-8a0c-421f-9693-b3ee90c448b9-root\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.176732 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.176684    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.178015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.177958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-tls\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.178015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.177973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27f3633c-8a0c-421f-9693-b3ee90c448b9-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.178617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.178596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.178744 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.178722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/84b77d03-d1fe-423d-91c1-3861873b302a-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.188803 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.188780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtjtw\" (UniqueName: \"kubernetes.io/projected/84b77d03-d1fe-423d-91c1-3861873b302a-kube-api-access-jtjtw\") pod \"kube-state-metrics-56d9f944b4-5q5w4\" (UID: \"84b77d03-d1fe-423d-91c1-3861873b302a\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.189256 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.189233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8n92\" (UniqueName: \"kubernetes.io/projected/27f3633c-8a0c-421f-9693-b3ee90c448b9-kube-api-access-b8n92\") pod \"node-exporter-84wmf\" (UID: \"27f3633c-8a0c-421f-9693-b3ee90c448b9\") " pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.260306 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.260274    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"
Jul 29 21:17:57.283612 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.283578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-84wmf"
Jul 29 21:17:57.964393 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:57.964356    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:17:58.063417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.063375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:17:58.063594 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.063508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.065922 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.065896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 21:17:58.066082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.065896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 21:17:58.066990 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.066967    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 21:17:58.067121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.067095    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 21:17:58.067198 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.067135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 21:17:58.067198 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.067178    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 21:17:58.067493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.067473    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 21:17:58.068240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.068222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 21:17:58.068351 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.068250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 21:17:58.068481 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.068462    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hw5x6\""
Jul 29 21:17:58.186165 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186130    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4n8\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186165 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186168    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186482    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186545    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186754 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186572    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.186754 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.186610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287490 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4n8\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287490 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287454    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287490 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287483    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.287979 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.288227 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.287988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.288227 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.288027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.288385 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.288364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.288492 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.288381    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.288584 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.288563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.290863 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.290840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.290963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.290898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291071 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291124 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291124 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291205 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.291790 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.291719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.292295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.292278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.297817 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.297799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4n8\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8\") pod \"alertmanager-main-0\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:58.376478 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:58.376434    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:17:59.453250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:17:59.453224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:17:59.861274 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:17:59.861220    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f3633c_8a0c_421f_9693_b3ee90c448b9.slice/crio-d8a0f56d826b5fb37d7f3db22c387a969c56155cd82bf7b5f8531c78dbfcd6ff WatchSource:0}: Error finding container d8a0f56d826b5fb37d7f3db22c387a969c56155cd82bf7b5f8531c78dbfcd6ff: Status 404 returned error can't find the container with id d8a0f56d826b5fb37d7f3db22c387a969c56155cd82bf7b5f8531c78dbfcd6ff
Jul 29 21:18:00.037030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.036878    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4"]
Jul 29 21:18:00.039125 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:18:00.039097    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b77d03_d1fe_423d_91c1_3861873b302a.slice/crio-ec676abba4f86d9625c8f9c2b17fc793ea758324297f712e700480cbbe472dde WatchSource:0}: Error finding container ec676abba4f86d9625c8f9c2b17fc793ea758324297f712e700480cbbe472dde: Status 404 returned error can't find the container with id ec676abba4f86d9625c8f9c2b17fc793ea758324297f712e700480cbbe472dde
Jul 29 21:18:00.258727 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.258693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:18:00.260865 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:18:00.260840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4208da79_b45f_4c81_9830_4ac21fcc569d.slice/crio-f76872ea35a2a5f68beb94f93209f4254ba55dbf7b9298defdcb800c1255b149 WatchSource:0}: Error finding container f76872ea35a2a5f68beb94f93209f4254ba55dbf7b9298defdcb800c1255b149: Status 404 returned error can't find the container with id f76872ea35a2a5f68beb94f93209f4254ba55dbf7b9298defdcb800c1255b149
Jul 29 21:18:00.268835 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.268812    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:18:00.270390 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:18:00.270369    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bad1590_5d81_4c97_a56f_b78d3361a79f.slice/crio-0ef80892b7cd73e24f7799e3fac9b41a6a5496936a80383e974929504069aa4a WatchSource:0}: Error finding container 0ef80892b7cd73e24f7799e3fac9b41a6a5496936a80383e974929504069aa4a: Status 404 returned error can't find the container with id 0ef80892b7cd73e24f7799e3fac9b41a6a5496936a80383e974929504069aa4a
Jul 29 21:18:00.565072 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.564958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-84wmf" event={"ID":"27f3633c-8a0c-421f-9693-b3ee90c448b9","Type":"ContainerStarted","Data":"d8a0f56d826b5fb37d7f3db22c387a969c56155cd82bf7b5f8531c78dbfcd6ff"}
Jul 29 21:18:00.567636 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.567551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccf7494b4-pvqtg" event={"ID":"94381045-f26f-415a-bca1-48b4c6127924","Type":"ContainerStarted","Data":"d58d8c7da625fb0f2344faaf65de77197c5dacea5e2ae29010a4766f573dd391"}
Jul 29 21:18:00.569359 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.569321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"0ef80892b7cd73e24f7799e3fac9b41a6a5496936a80383e974929504069aa4a"}
Jul 29 21:18:00.570794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.570759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4" event={"ID":"84b77d03-d1fe-423d-91c1-3861873b302a","Type":"ContainerStarted","Data":"ec676abba4f86d9625c8f9c2b17fc793ea758324297f712e700480cbbe472dde"}
Jul 29 21:18:00.572782 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.572749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb9cd67f7-mbvfs" event={"ID":"4208da79-b45f-4c81-9830-4ac21fcc569d","Type":"ContainerStarted","Data":"48b386f0a9d9791c9b5fac0bbd02479c3fa92836a52184fb61a09e1dd80bd113"}
Jul 29 21:18:00.572884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.572802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb9cd67f7-mbvfs" event={"ID":"4208da79-b45f-4c81-9830-4ac21fcc569d","Type":"ContainerStarted","Data":"f76872ea35a2a5f68beb94f93209f4254ba55dbf7b9298defdcb800c1255b149"}
Jul 29 21:18:00.574655 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.574624    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-6wc8p" event={"ID":"d83b763d-5d3c-42c7-bd05-b7b3992a1e81","Type":"ContainerStarted","Data":"bbd3d53614f86d9664e1d62208902c4e0a081f421a14e7c9349805fc78eda327"}
Jul 29 21:18:00.574993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.574968    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:18:00.587515 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.586775    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ccf7494b4-pvqtg" podStartSLOduration=2.389480449 podStartE2EDuration="12.586754345s" totalImagesPullingTime="10.197273896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:00.585518826 +0000 UTC m=+97.057097923" watchObservedRunningTime="2026-07-29 21:18:00.586754345 +0000 UTC m=+97.058333422"
Jul 29 21:18:00.589642 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.589615    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-6wc8p"
Jul 29 21:18:00.604499 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.604436    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fb9cd67f7-mbvfs" podStartSLOduration=6.60441577 podStartE2EDuration="6.60441577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:00.602742732 +0000 UTC m=+97.074321811" watchObservedRunningTime="2026-07-29 21:18:00.60441577 +0000 UTC m=+97.075994910"
Jul 29 21:18:00.620663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:00.620614    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-6wc8p" podStartSLOduration=2.420456429 podStartE2EDuration="22.620603284s" totalImagesPullingTime="20.200146855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:00.619783292 +0000 UTC m=+97.091362367" watchObservedRunningTime="2026-07-29 21:18:00.620603284 +0000 UTC m=+97.092182360"
Jul 29 21:18:01.604745 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.604655    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:18:01.621775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.621664    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"]
Jul 29 21:18:01.644659 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.643511    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"]
Jul 29 21:18:01.644865 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.643911    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.649680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.649180    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-tfgqx\""
Jul 29 21:18:01.649680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.649585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 29 21:18:01.822058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.821736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa80b862-1c98-4243-85dd-26ba90602d0d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.822058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.821871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkkz\" (UniqueName: \"kubernetes.io/projected/aa80b862-1c98-4243-85dd-26ba90602d0d-kube-api-access-ntkkz\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.923082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.922519    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkkz\" (UniqueName: \"kubernetes.io/projected/aa80b862-1c98-4243-85dd-26ba90602d0d-kube-api-access-ntkkz\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.923082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.922601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa80b862-1c98-4243-85dd-26ba90602d0d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.935374 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.935313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkkz\" (UniqueName: \"kubernetes.io/projected/aa80b862-1c98-4243-85dd-26ba90602d0d-kube-api-access-ntkkz\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.941590 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.941553    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/aa80b862-1c98-4243-85dd-26ba90602d0d-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-7dzzt\" (UID: \"aa80b862-1c98-4243-85dd-26ba90602d0d\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:01.963108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:01.963072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:03.043843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.043430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"]
Jul 29 21:18:03.044711 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:18:03.044652    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa80b862_1c98_4243_85dd_26ba90602d0d.slice/crio-ad57c0ed303b0a3085eb84661dee64ebd754a8747bf1ba7c597b8ff8d3ea647f WatchSource:0}: Error finding container ad57c0ed303b0a3085eb84661dee64ebd754a8747bf1ba7c597b8ff8d3ea647f: Status 404 returned error can't find the container with id ad57c0ed303b0a3085eb84661dee64ebd754a8747bf1ba7c597b8ff8d3ea647f
Jul 29 21:18:03.057508 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.057178    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:18:03.075176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.075152    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.079445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.077990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t6r44\""
Jul 29 21:18:03.079445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.078272    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 21:18:03.079445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.078646    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 21:18:03.079445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.078877    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 21:18:03.079445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.079234    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 21:18:03.080998 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.080711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.082884    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.082916    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.083160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.083355    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.083599    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qsdqkjdipadp\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.085786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.086012    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.086232    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.086829    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 21:18:03.089937 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.087223    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 21:18:03.092700 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.092315    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 21:18:03.097936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.096670    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:18:03.237361 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237573 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237573 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237573 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237573 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237541    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xl4k\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.237794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237878    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237942    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237967    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.237997    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.238027    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.238103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.238102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xl4k\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339757    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.339839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.339795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.340488 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.340171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.342387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.341082    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.342387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.341731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.342387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.342034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.342587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.342565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.343340 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.343007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.343340 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.343115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.343340 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.343161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.343678 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.343654    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.344010 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.343981    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.344525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.344501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.345255 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.345213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.346409 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.346382    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.346562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.346541    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.347007 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.346693    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.347007 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.346966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.347414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.347172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.347928 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.347911    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.348554 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.348536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.351767 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.351746    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xl4k\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k\") pod \"prometheus-k8s-0\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.500372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.500325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:03.595265 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.594459    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6" exitCode=0
Jul 29 21:18:03.595265 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.594556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"}
Jul 29 21:18:03.602353 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.602322    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4" event={"ID":"84b77d03-d1fe-423d-91c1-3861873b302a","Type":"ContainerStarted","Data":"d757febdf668d9fbdbc32bcc153767d71a66a78c09a6d2747d15980220ca2da8"}
Jul 29 21:18:03.603511 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.603481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4" event={"ID":"84b77d03-d1fe-423d-91c1-3861873b302a","Type":"ContainerStarted","Data":"8ef11b08cae3dec97143497eeb9626a7b20c5d92505c09b78acd24778d3a926d"}
Jul 29 21:18:03.603691 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.603662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4" event={"ID":"84b77d03-d1fe-423d-91c1-3861873b302a","Type":"ContainerStarted","Data":"de24f54768faea90d0637cdabbd1e6268ebafe45b67d66583820c7ad6ccb4d06"}
Jul 29 21:18:03.605315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.605284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt" event={"ID":"aa80b862-1c98-4243-85dd-26ba90602d0d","Type":"ContainerStarted","Data":"ad57c0ed303b0a3085eb84661dee64ebd754a8747bf1ba7c597b8ff8d3ea647f"}
Jul 29 21:18:03.607550 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.607527    2565 generic.go:356] "Generic (PLEG): container finished" podID="27f3633c-8a0c-421f-9693-b3ee90c448b9" containerID="533529a2efa5c149fe9f2e2ebdf5d8a519c9fbc9f2b4fb793406a0af787213fc" exitCode=0
Jul 29 21:18:03.607682 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.607657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-84wmf" event={"ID":"27f3633c-8a0c-421f-9693-b3ee90c448b9","Type":"ContainerDied","Data":"533529a2efa5c149fe9f2e2ebdf5d8a519c9fbc9f2b4fb793406a0af787213fc"}
Jul 29 21:18:03.681542 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.681482    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-5q5w4" podStartSLOduration=4.852145574 podStartE2EDuration="7.681455166s" totalImagesPullingTime="2.829309592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:03.678549634 +0000 UTC m=+100.150128710" watchObservedRunningTime="2026-07-29 21:18:03.681455166 +0000 UTC m=+100.153034243"
Jul 29 21:18:03.718297 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.718226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:18:03.721173 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:18:03.721141    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b76f3db_0c86_4f89_a131_503dd2438fe1.slice/crio-5cb22173a1597a91126c2ce9aca205358401a1068801c6305f38202211ba3fb5 WatchSource:0}: Error finding container 5cb22173a1597a91126c2ce9aca205358401a1068801c6305f38202211ba3fb5: Status 404 returned error can't find the container with id 5cb22173a1597a91126c2ce9aca205358401a1068801c6305f38202211ba3fb5
Jul 29 21:18:03.856637 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:03.856601    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:18:04.614670 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.614609    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76" exitCode=0
Jul 29 21:18:04.615283 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.614692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"}
Jul 29 21:18:04.615283 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.614766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"5cb22173a1597a91126c2ce9aca205358401a1068801c6305f38202211ba3fb5"}
Jul 29 21:18:04.617766 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.617739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-84wmf" event={"ID":"27f3633c-8a0c-421f-9693-b3ee90c448b9","Type":"ContainerStarted","Data":"6829d1f7e7926cda9cd3593353cb3647c5aa269b16b8e85250f047ccc4390707"}
Jul 29 21:18:04.617871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.617791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-84wmf" event={"ID":"27f3633c-8a0c-421f-9693-b3ee90c448b9","Type":"ContainerStarted","Data":"e14c9a5ce624355769515300499887970dc8a8b61577edefbc26e3c35d3c352e"}
Jul 29 21:18:04.855293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.854925    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:04.855475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.855200    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:04.862863 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.862245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:04.885263 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:04.884780    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-84wmf" podStartSLOduration=7.148197265 podStartE2EDuration="8.88476084s" totalImagesPullingTime="1.617176833s" totalInitContainerRuntime="119.386742ms" isStatefulPod=true podCreationTimestamp="2026-07-29 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:04.697407489 +0000 UTC m=+101.168986589" watchObservedRunningTime="2026-07-29 21:18:04.88476084 +0000 UTC m=+101.356339935"
Jul 29 21:18:05.627494 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:05.627464    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:06.636101 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.635831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"}
Jul 29 21:18:06.636101 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.635922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"}
Jul 29 21:18:06.636101 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.635966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"}
Jul 29 21:18:06.639776 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.639699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt" event={"ID":"aa80b862-1c98-4243-85dd-26ba90602d0d","Type":"ContainerStarted","Data":"0f497f811edf33576b431a8c06ab85f53659b0daa6d6fd4fcbe9d968b77f7b21"}
Jul 29 21:18:06.646942 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.646614    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:06.654237 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.653722    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt"
Jul 29 21:18:06.665446 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:06.665356    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-7dzzt" podStartSLOduration=2.558853355 podStartE2EDuration="5.665335276s" totalImagesPullingTime="3.106481921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:06.664850947 +0000 UTC m=+103.136430024" watchObservedRunningTime="2026-07-29 21:18:06.665335276 +0000 UTC m=+103.136914355"
Jul 29 21:18:07.652696 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:07.652305    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"}
Jul 29 21:18:07.652696 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:07.652438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"}
Jul 29 21:18:08.556679 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:08.556637    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:18:09.451113 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:09.451066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pm5pq"
Jul 29 21:18:10.668210 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:10.668175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerStarted","Data":"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"}
Jul 29 21:18:10.670233 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:10.670211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"}
Jul 29 21:18:10.670334 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:10.670264    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"}
Jul 29 21:18:10.701611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:10.701559    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.698195967 podStartE2EDuration="13.701543985s" totalImagesPullingTime="7.962656583s" totalInitContainerRuntime="40.691435ms" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:10.699093455 +0000 UTC m=+107.170672531" watchObservedRunningTime="2026-07-29 21:18:10.701543985 +0000 UTC m=+107.173123059"
Jul 29 21:18:12.580428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:12.580389    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:18:13.685074 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:13.685023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"}
Jul 29 21:18:13.685545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:13.685112    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"}
Jul 29 21:18:13.685545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:13.685144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"}
Jul 29 21:18:13.685545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:13.685172    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerStarted","Data":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"}
Jul 29 21:18:13.735175 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:13.735110    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.860552197 podStartE2EDuration="10.735087175s" totalImagesPullingTime="7.841742238s" totalInitContainerRuntime="32.79274ms" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:13.732443628 +0000 UTC m=+110.204022727" watchObservedRunningTime="2026-07-29 21:18:13.735087175 +0000 UTC m=+110.206666250"
Jul 29 21:18:18.501029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:18.500993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:18:27.606172 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.606108    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ccf7494b4-pvqtg" podUID="94381045-f26f-415a-bca1-48b4c6127924" containerName="console" containerID="cri-o://d58d8c7da625fb0f2344faaf65de77197c5dacea5e2ae29010a4766f573dd391" gracePeriod=15
Jul 29 21:18:27.738468 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.738427    2565 generic.go:356] "Generic (PLEG): container finished" podID="94381045-f26f-415a-bca1-48b4c6127924" containerID="d58d8c7da625fb0f2344faaf65de77197c5dacea5e2ae29010a4766f573dd391" exitCode=0
Jul 29 21:18:27.738670 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.738499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccf7494b4-pvqtg" event={"ID":"94381045-f26f-415a-bca1-48b4c6127924","Type":"ContainerDied","Data":"d58d8c7da625fb0f2344faaf65de77197c5dacea5e2ae29010a4766f573dd391"}
Jul 29 21:18:27.875004 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.874931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:18:27.967926 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.967898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.967949    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968023    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968038    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968092    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96cbh\" (UniqueName: \"kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh\") pod \"94381045-f26f-415a-bca1-48b4c6127924\" (UID: \"94381045-f26f-415a-bca1-48b4c6127924\") "
Jul 29 21:18:27.968371 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968343    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:27.968460 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968429    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config" (OuterVolumeSpecName: "console-config") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:27.968460 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.968431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca" (OuterVolumeSpecName: "service-ca") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:27.970308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.970276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:27.970389 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.970318    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:27.970389 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:27.970334    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh" (OuterVolumeSpecName: "kube-api-access-96cbh") pod "94381045-f26f-415a-bca1-48b4c6127924" (UID: "94381045-f26f-415a-bca1-48b4c6127924"). InnerVolumeSpecName "kube-api-access-96cbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:28.069693 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069666    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-console-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.069693 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069693    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.069851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069702    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-serving-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.069851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069711    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96cbh\" (UniqueName: \"kubernetes.io/projected/94381045-f26f-415a-bca1-48b4c6127924-kube-api-access-96cbh\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.069851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069720    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/94381045-f26f-415a-bca1-48b4c6127924-oauth-serving-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.069851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.069728    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/94381045-f26f-415a-bca1-48b4c6127924-console-oauth-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:28.743512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.743482    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ccf7494b4-pvqtg"
Jul 29 21:18:28.743512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.743499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ccf7494b4-pvqtg" event={"ID":"94381045-f26f-415a-bca1-48b4c6127924","Type":"ContainerDied","Data":"7e8bc74d6d37b925ce5a5e9be98537fc2b5b892e6e1c4da00f3e19652edd24e8"}
Jul 29 21:18:28.744006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.743553    2565 scope.go:117] "RemoveContainer" containerID="d58d8c7da625fb0f2344faaf65de77197c5dacea5e2ae29010a4766f573dd391"
Jul 29 21:18:28.760379 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.760348    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:18:28.763922 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.763892    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ccf7494b4-pvqtg"]
Jul 29 21:18:28.880132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:28.880066    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerName="registry" containerID="cri-o://55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600" gracePeriod=30
Jul 29 21:18:29.159620 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.159589    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:18:29.278263 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278263 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278244    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278268    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278426    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278642 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278642 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdqrx\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx\") pod \"25296465-f81c-4654-8bdf-6f6edb8e1418\" (UID: \"25296465-f81c-4654-8bdf-6f6edb8e1418\") "
Jul 29 21:18:29.278737 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:29.278921 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.278883    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-trusted-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.279099 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.279072    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:29.280884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.280852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:29.281142 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.281112    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:29.281241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.281152    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:29.281293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.281233    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:29.281293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.281236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx" (OuterVolumeSpecName: "kube-api-access-hdqrx") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "kube-api-access-hdqrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:29.287164 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.287140    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "25296465-f81c-4654-8bdf-6f6edb8e1418" (UID: "25296465-f81c-4654-8bdf-6f6edb8e1418"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:18:29.379468 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379428    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/25296465-f81c-4654-8bdf-6f6edb8e1418-ca-trust-extracted\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379468 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379461    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-installation-pull-secrets\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379468 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379474    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/25296465-f81c-4654-8bdf-6f6edb8e1418-image-registry-private-configuration\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379485    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-bound-sa-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379501    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379511    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/25296465-f81c-4654-8bdf-6f6edb8e1418-registry-certificates\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.379699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.379524    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdqrx\" (UniqueName: \"kubernetes.io/projected/25296465-f81c-4654-8bdf-6f6edb8e1418-kube-api-access-hdqrx\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:29.748524 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.748490    2565 generic.go:356] "Generic (PLEG): container finished" podID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerID="55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600" exitCode=0
Jul 29 21:18:29.748991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.748535    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t"
Jul 29 21:18:29.748991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.748577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" event={"ID":"25296465-f81c-4654-8bdf-6f6edb8e1418","Type":"ContainerDied","Data":"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"}
Jul 29 21:18:29.748991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.748634    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b854cf867-9jd8t" event={"ID":"25296465-f81c-4654-8bdf-6f6edb8e1418","Type":"ContainerDied","Data":"a39352e0b4a08182c8b92ebc93d9b0de4bf515e7ea057f8719e970fef22f685c"}
Jul 29 21:18:29.748991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.748667    2565 scope.go:117] "RemoveContainer" containerID="55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"
Jul 29 21:18:29.757548 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.757512    2565 scope.go:117] "RemoveContainer" containerID="55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"
Jul 29 21:18:29.775193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.775099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:18:29.779969 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:18:29.779873    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600\": container with ID starting with 55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600 not found: ID does not exist" containerID="55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"
Jul 29 21:18:29.779969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.779934    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600"} err="failed to get container status \"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600\": rpc error: code = NotFound desc = could not find container \"55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600\": container with ID starting with 55a3e7c78d5a6130f068dd279cfef46d7449f22f5b60ededb8ce8359d5231600 not found: ID does not exist"
Jul 29 21:18:29.781331 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:29.781307    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b854cf867-9jd8t"]
Jul 29 21:18:30.054134 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:30.054034    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" path="/var/lib/kubelet/pods/25296465-f81c-4654-8bdf-6f6edb8e1418/volumes"
Jul 29 21:18:30.054452 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:30.054439    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94381045-f26f-415a-bca1-48b4c6127924" path="/var/lib/kubelet/pods/94381045-f26f-415a-bca1-48b4c6127924/volumes"
Jul 29 21:18:37.599191 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.599148    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fb9cd67f7-mbvfs" podUID="4208da79-b45f-4c81-9830-4ac21fcc569d" containerName="console" containerID="cri-o://48b386f0a9d9791c9b5fac0bbd02479c3fa92836a52184fb61a09e1dd80bd113" gracePeriod=15
Jul 29 21:18:37.782672 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.782636    2565 generic.go:356] "Generic (PLEG): container finished" podID="4208da79-b45f-4c81-9830-4ac21fcc569d" containerID="48b386f0a9d9791c9b5fac0bbd02479c3fa92836a52184fb61a09e1dd80bd113" exitCode=0
Jul 29 21:18:37.782836 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.782698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb9cd67f7-mbvfs" event={"ID":"4208da79-b45f-4c81-9830-4ac21fcc569d","Type":"ContainerDied","Data":"48b386f0a9d9791c9b5fac0bbd02479c3fa92836a52184fb61a09e1dd80bd113"}
Jul 29 21:18:37.840294 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.840268    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:37.948057 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948004    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgpx7\" (UniqueName: \"kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948451 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948237    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config\") pod \"4208da79-b45f-4c81-9830-4ac21fcc569d\" (UID: \"4208da79-b45f-4c81-9830-4ac21fcc569d\") "
Jul 29 21:18:37.948663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948548    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:37.948663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca" (OuterVolumeSpecName: "service-ca") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:37.948795 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948765    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config" (OuterVolumeSpecName: "console-config") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:37.948885 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.948867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:18:37.950506 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.950468    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:37.950595 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.950549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:18:37.950595 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:37.950559    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7" (OuterVolumeSpecName: "kube-api-access-lgpx7") pod "4208da79-b45f-4c81-9830-4ac21fcc569d" (UID: "4208da79-b45f-4c81-9830-4ac21fcc569d"). InnerVolumeSpecName "kube-api-access-lgpx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:18:38.049209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049171    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-oauth-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049204    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049214    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4208da79-b45f-4c81-9830-4ac21fcc569d-console-serving-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049223    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-oauth-serving-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049231    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-trusted-ca-bundle\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049239    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4208da79-b45f-4c81-9830-4ac21fcc569d-console-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.049429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.049247    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgpx7\" (UniqueName: \"kubernetes.io/projected/4208da79-b45f-4c81-9830-4ac21fcc569d-kube-api-access-lgpx7\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:18:38.788677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.788633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fb9cd67f7-mbvfs" event={"ID":"4208da79-b45f-4c81-9830-4ac21fcc569d","Type":"ContainerDied","Data":"f76872ea35a2a5f68beb94f93209f4254ba55dbf7b9298defdcb800c1255b149"}
Jul 29 21:18:38.789138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.788718    2565 scope.go:117] "RemoveContainer" containerID="48b386f0a9d9791c9b5fac0bbd02479c3fa92836a52184fb61a09e1dd80bd113"
Jul 29 21:18:38.789138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.788658    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fb9cd67f7-mbvfs"
Jul 29 21:18:38.806407 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.806383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:18:38.812644 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:38.812613    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fb9cd67f7-mbvfs"]
Jul 29 21:18:40.054996 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:18:40.054966    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4208da79-b45f-4c81-9830-4ac21fcc569d" path="/var/lib/kubelet/pods/4208da79-b45f-4c81-9830-4ac21fcc569d/volumes"
Jul 29 21:19:03.501314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:03.501257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:03.520182 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:03.520153    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:03.905672 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:03.905594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:17.170969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.170883    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171220    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="alertmanager" containerID="cri-o://f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d" gracePeriod=120
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171297    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-metric" containerID="cri-o://bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f" gracePeriod=120
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171323    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="config-reloader" containerID="cri-o://f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da" gracePeriod=120
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171359    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy" containerID="cri-o://9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe" gracePeriod=120
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171386    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="prom-label-proxy" containerID="cri-o://ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad" gracePeriod=120
Jul 29 21:19:17.171475 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.171305    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-web" containerID="cri-o://0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e" gracePeriod=120
Jul 29 21:19:17.951788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951755    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad" exitCode=0
Jul 29 21:19:17.951788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951782    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe" exitCode=0
Jul 29 21:19:17.951788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951788    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da" exitCode=0
Jul 29 21:19:17.951788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951794    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d" exitCode=0
Jul 29 21:19:17.952066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951825    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"}
Jul 29 21:19:17.952066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951921    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"}
Jul 29 21:19:17.952066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951949    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"}
Jul 29 21:19:17.952066 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:17.951973    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"}
Jul 29 21:19:18.415135 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.415110    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:18.480212 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480177    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480235    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480263    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480387 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480458    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.480673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480656    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:18.480919 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.480802    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-metrics-client-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.481331 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.481285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:18.483535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.483490    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.483839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.483813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out" (OuterVolumeSpecName: "config-out") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:19:18.484128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.484081    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.484230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.484130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.484230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.484195    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume" (OuterVolumeSpecName: "config-volume") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.484788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.484738    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.488648 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.488492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.495526 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.495495    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config" (OuterVolumeSpecName: "web-config") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:18.581565 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581511    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr4n8\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.581565 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581588    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db\") pod \"1bad1590-5d81-4c97-a56f-b78d3361a79f\" (UID: \"1bad1590-5d81-4c97-a56f-b78d3361a79f\") "
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581711    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-out\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581722    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581733    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-config-volume\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581749    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581760    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-cluster-tls-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581770    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-web-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581967 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581779    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581967 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581788    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1bad1590-5d81-4c97-a56f-b78d3361a79f-secret-alertmanager-main-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.581967 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581796    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.582079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.581973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:19:18.583579 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.583544    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:19:18.583663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.583588    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8" (OuterVolumeSpecName: "kube-api-access-gr4n8") pod "1bad1590-5d81-4c97-a56f-b78d3361a79f" (UID: "1bad1590-5d81-4c97-a56f-b78d3361a79f"). InnerVolumeSpecName "kube-api-access-gr4n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:19:18.682336 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.682301    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4n8\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-kube-api-access-gr4n8\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.682336 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.682333    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1bad1590-5d81-4c97-a56f-b78d3361a79f-tls-assets\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.682336 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.682346    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1bad1590-5d81-4c97-a56f-b78d3361a79f-alertmanager-main-db\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:18.959539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959499    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f" exitCode=0
Jul 29 21:19:18.959539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959526    2565 generic.go:356] "Generic (PLEG): container finished" podID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerID="0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e" exitCode=0
Jul 29 21:19:18.959748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:18.959748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"}
Jul 29 21:19:18.959748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"}
Jul 29 21:19:18.959748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1bad1590-5d81-4c97-a56f-b78d3361a79f","Type":"ContainerDied","Data":"0ef80892b7cd73e24f7799e3fac9b41a6a5496936a80383e974929504069aa4a"}
Jul 29 21:19:18.959748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.959715    2565 scope.go:117] "RemoveContainer" containerID="ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"
Jul 29 21:19:18.967436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.967408    2565 scope.go:117] "RemoveContainer" containerID="bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"
Jul 29 21:19:18.976748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.976723    2565 scope.go:117] "RemoveContainer" containerID="9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"
Jul 29 21:19:18.983843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.983822    2565 scope.go:117] "RemoveContainer" containerID="0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"
Jul 29 21:19:18.989684 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.989654    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:18.992061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.992033    2565 scope.go:117] "RemoveContainer" containerID="f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"
Jul 29 21:19:18.999574 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:18.999552    2565 scope.go:117] "RemoveContainer" containerID="f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"
Jul 29 21:19:19.000236 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.000212    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:19.006787 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.006762    2565 scope.go:117] "RemoveContainer" containerID="75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"
Jul 29 21:19:19.013721 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.013700    2565 scope.go:117] "RemoveContainer" containerID="ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"
Jul 29 21:19:19.013986 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.013967    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad\": container with ID starting with ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad not found: ID does not exist" containerID="ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"
Jul 29 21:19:19.014031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.013994    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"} err="failed to get container status \"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad\": rpc error: code = NotFound desc = could not find container \"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad\": container with ID starting with ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad not found: ID does not exist"
Jul 29 21:19:19.014031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014014    2565 scope.go:117] "RemoveContainer" containerID="bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"
Jul 29 21:19:19.014276 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.014257    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f\": container with ID starting with bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f not found: ID does not exist" containerID="bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"
Jul 29 21:19:19.014316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014283    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"} err="failed to get container status \"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f\": rpc error: code = NotFound desc = could not find container \"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f\": container with ID starting with bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f not found: ID does not exist"
Jul 29 21:19:19.014316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014302    2565 scope.go:117] "RemoveContainer" containerID="9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"
Jul 29 21:19:19.014508 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.014491    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe\": container with ID starting with 9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe not found: ID does not exist" containerID="9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"
Jul 29 21:19:19.014546 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014517    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"} err="failed to get container status \"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe\": rpc error: code = NotFound desc = could not find container \"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe\": container with ID starting with 9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe not found: ID does not exist"
Jul 29 21:19:19.014546 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014531    2565 scope.go:117] "RemoveContainer" containerID="0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"
Jul 29 21:19:19.014744 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.014729    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e\": container with ID starting with 0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e not found: ID does not exist" containerID="0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"
Jul 29 21:19:19.014779 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014749    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"} err="failed to get container status \"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e\": rpc error: code = NotFound desc = could not find container \"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e\": container with ID starting with 0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e not found: ID does not exist"
Jul 29 21:19:19.014779 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014762    2565 scope.go:117] "RemoveContainer" containerID="f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"
Jul 29 21:19:19.014994 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.014979    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da\": container with ID starting with f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da not found: ID does not exist" containerID="f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"
Jul 29 21:19:19.015135 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.014997    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"} err="failed to get container status \"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da\": rpc error: code = NotFound desc = could not find container \"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da\": container with ID starting with f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da not found: ID does not exist"
Jul 29 21:19:19.015135 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015013    2565 scope.go:117] "RemoveContainer" containerID="f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"
Jul 29 21:19:19.015251 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.015236    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d\": container with ID starting with f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d not found: ID does not exist" containerID="f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"
Jul 29 21:19:19.015287 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015253    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"} err="failed to get container status \"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d\": rpc error: code = NotFound desc = could not find container \"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d\": container with ID starting with f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d not found: ID does not exist"
Jul 29 21:19:19.015287 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015265    2565 scope.go:117] "RemoveContainer" containerID="75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"
Jul 29 21:19:19.015490 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:19.015475    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6\": container with ID starting with 75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6 not found: ID does not exist" containerID="75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"
Jul 29 21:19:19.015537 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015494    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"} err="failed to get container status \"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6\": rpc error: code = NotFound desc = could not find container \"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6\": container with ID starting with 75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6 not found: ID does not exist"
Jul 29 21:19:19.015537 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015506    2565 scope.go:117] "RemoveContainer" containerID="ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"
Jul 29 21:19:19.015721 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015693    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad"} err="failed to get container status \"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad\": rpc error: code = NotFound desc = could not find container \"ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad\": container with ID starting with ba08ebe81ae4e1fa53df8bd1614b266bcf61f292feee9e23e8e52c1a5c4c9fad not found: ID does not exist"
Jul 29 21:19:19.015763 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015722    2565 scope.go:117] "RemoveContainer" containerID="bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"
Jul 29 21:19:19.015925 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015910    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f"} err="failed to get container status \"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f\": rpc error: code = NotFound desc = could not find container \"bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f\": container with ID starting with bd1abec816bfba9a5d720af9e78627a1eafe2e712c296ccab569a921c568a68f not found: ID does not exist"
Jul 29 21:19:19.016019 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.015926    2565 scope.go:117] "RemoveContainer" containerID="9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"
Jul 29 21:19:19.016134 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016112    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe"} err="failed to get container status \"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe\": rpc error: code = NotFound desc = could not find container \"9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe\": container with ID starting with 9c88f0b33d337e4ab70917a118d5beabfe982ae71b1ec8b40db864c66a5748fe not found: ID does not exist"
Jul 29 21:19:19.016182 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016136    2565 scope.go:117] "RemoveContainer" containerID="0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"
Jul 29 21:19:19.016348 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016333    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e"} err="failed to get container status \"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e\": rpc error: code = NotFound desc = could not find container \"0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e\": container with ID starting with 0732cc41a2c45b327af320ba5822ac18b5a9eb74fb8eb937775663ffda0aee7e not found: ID does not exist"
Jul 29 21:19:19.016348 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016348    2565 scope.go:117] "RemoveContainer" containerID="f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"
Jul 29 21:19:19.016549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016535    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da"} err="failed to get container status \"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da\": rpc error: code = NotFound desc = could not find container \"f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da\": container with ID starting with f83024a1443c29b7adfa6ba6e81cd5a6a819afe7a551c9fed455202d5ae536da not found: ID does not exist"
Jul 29 21:19:19.016592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016550    2565 scope.go:117] "RemoveContainer" containerID="f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"
Jul 29 21:19:19.016803 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016785    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d"} err="failed to get container status \"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d\": rpc error: code = NotFound desc = could not find container \"f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d\": container with ID starting with f9059c5cb6b324a33f6a98efebeb38d6f4ff8dcdc5c8ab04c6c2f1dffd62b76d not found: ID does not exist"
Jul 29 21:19:19.016845 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016805    2565 scope.go:117] "RemoveContainer" containerID="75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"
Jul 29 21:19:19.017011 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.016994    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6"} err="failed to get container status \"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6\": rpc error: code = NotFound desc = could not find container \"75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6\": container with ID starting with 75f7b20d84d02a682aafd5c43aa77915cbe26856542601a0b7f8b0fc9a8a24d6 not found: ID does not exist"
Jul 29 21:19:19.031861 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.031827    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:19.032125 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032112    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4208da79-b45f-4c81-9830-4ac21fcc569d" containerName="console"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032127    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4208da79-b45f-4c81-9830-4ac21fcc569d" containerName="console"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="alertmanager"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="alertmanager"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032151    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-web"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032156    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-web"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032164    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032169    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy"
Jul 29 21:19:19.032174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032175    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="prom-label-proxy"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032180    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="prom-label-proxy"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032191    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94381045-f26f-415a-bca1-48b4c6127924" containerName="console"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032196    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="94381045-f26f-415a-bca1-48b4c6127924" containerName="console"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032203    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="config-reloader"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032208    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="config-reloader"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032215    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerName="registry"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032220    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerName="registry"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032231    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-metric"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032236    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-metric"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032243    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="init-config-reloader"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032248    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="init-config-reloader"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032286    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-metric"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032293    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy-web"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032299    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="prom-label-proxy"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032306    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="config-reloader"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032314    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="94381045-f26f-415a-bca1-48b4c6127924" containerName="console"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032321    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="alertmanager"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032326    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4208da79-b45f-4c81-9830-4ac21fcc569d" containerName="console"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032331    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" containerName="kube-rbac-proxy"
Jul 29 21:19:19.032390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.032335    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25296465-f81c-4654-8bdf-6f6edb8e1418" containerName="registry"
Jul 29 21:19:19.037164 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.037145    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.039830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.039805    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 29 21:19:19.039949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.039805    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hw5x6\""
Jul 29 21:19:19.039949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.039804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 29 21:19:19.040054 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.039973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 29 21:19:19.040260 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.040243    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 29 21:19:19.040311 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.040294    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 29 21:19:19.040364 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.040306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 29 21:19:19.040364 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.040304    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 29 21:19:19.040592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.040575    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 29 21:19:19.047485 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.047464    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 29 21:19:19.052660 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.052636    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:19.085501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-web-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085710 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-config-out\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085756    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28z8\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-kube-api-access-x28z8\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.085911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.085814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186181 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-config-out\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186181 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28z8\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-kube-api-access-x28z8\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-web-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186360    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.186891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.186702    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.187689 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.187302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189083 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f3378642-6c50-455f-9d45-07c01a88bb20-config-out\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189331 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189497 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189490    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-web-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189698 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.189698 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.189691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f3378642-6c50-455f-9d45-07c01a88bb20-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.190210 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.190190    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.191214 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.191193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f3378642-6c50-455f-9d45-07c01a88bb20-config-volume\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.194240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.194223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28z8\" (UniqueName: \"kubernetes.io/projected/f3378642-6c50-455f-9d45-07c01a88bb20-kube-api-access-x28z8\") pod \"alertmanager-main-0\" (UID: \"f3378642-6c50-455f-9d45-07c01a88bb20\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.346339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.346299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 29 21:19:19.486980 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.486947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 29 21:19:19.491303 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:19:19.491273    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3378642_6c50_455f_9d45_07c01a88bb20.slice/crio-1fdfce4ae095a714d50baca0b67ab601bcdc2c9345fa7bacfe309c0c04ffc878 WatchSource:0}: Error finding container 1fdfce4ae095a714d50baca0b67ab601bcdc2c9345fa7bacfe309c0c04ffc878: Status 404 returned error can't find the container with id 1fdfce4ae095a714d50baca0b67ab601bcdc2c9345fa7bacfe309c0c04ffc878
Jul 29 21:19:19.965773 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.965735    2565 generic.go:356] "Generic (PLEG): container finished" podID="f3378642-6c50-455f-9d45-07c01a88bb20" containerID="e835de2e1289bc071f16f4c0ca42097f782e8aa93814c38dc83b381148883ce9" exitCode=0
Jul 29 21:19:19.965966 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.965828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerDied","Data":"e835de2e1289bc071f16f4c0ca42097f782e8aa93814c38dc83b381148883ce9"}
Jul 29 21:19:19.965966 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:19.965945    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"1fdfce4ae095a714d50baca0b67ab601bcdc2c9345fa7bacfe309c0c04ffc878"}
Jul 29 21:19:20.055103 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.055072    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bad1590-5d81-4c97-a56f-b78d3361a79f" path="/var/lib/kubelet/pods/1bad1590-5d81-4c97-a56f-b78d3361a79f/volumes"
Jul 29 21:19:20.974406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974374    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"2822365ae41d8012ae9548a38c30dcdd61cd3d19b8dd77784a283ddad72a32a1"}
Jul 29 21:19:20.974770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974438    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"b3bac3b6435959e301649f967587017965d7535801ab5c039000c8ae6f73beef"}
Jul 29 21:19:20.974770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"57af0a4fed1da90732f90549745d197083c3eb4c3d9d3adffc28efe0fd605144"}
Jul 29 21:19:20.974770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"76c61479d20af9353895975ba0c661617134230a039baff7653f0178bfd9afe9"}
Jul 29 21:19:20.974770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"08c373208ae13364569fbd395bb34d82f9dfc4215b6152ad95394cb542879146"}
Jul 29 21:19:20.974770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:20.974542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f3378642-6c50-455f-9d45-07c01a88bb20","Type":"ContainerStarted","Data":"8f30e8926d069786fbab4b2fb0e55ccfb5fc9af68c7e2c296c355b7b354eeae4"}
Jul 29 21:19:21.008394 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.008336    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.992505739 podStartE2EDuration="2.00831857s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.812831ms" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:21.002245563 +0000 UTC m=+177.473824637" watchObservedRunningTime="2026-07-29 21:19:21.00831857 +0000 UTC m=+177.479897646"
Jul 29 21:19:21.162274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.162239    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"]
Jul 29 21:19:21.165796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.165779    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.168338 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.168314    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 29 21:19:21.168597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.168578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-88x2x\""
Jul 29 21:19:21.168681 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.168667    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 29 21:19:21.168869 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.168850    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 29 21:19:21.168949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.168862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 29 21:19:21.169130 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.169110    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 29 21:19:21.175189 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.175168    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 29 21:19:21.185838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.185816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"]
Jul 29 21:19:21.202131 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202103    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202131 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202325 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4mh\" (UniqueName: \"kubernetes.io/projected/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-kube-api-access-6p4mh\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202325 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-serving-certs-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202325 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202286    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-metrics-client-ca\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202773 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202312    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-federate-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202773 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.202773 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.202618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-metrics-client-ca\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-federate-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303565    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303588    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.303911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4mh\" (UniqueName: \"kubernetes.io/projected/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-kube-api-access-6p4mh\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.304016 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.303920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-serving-certs-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.304281 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.304254    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-metrics-client-ca\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.304604 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.304578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.304699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.304610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-serving-certs-ca-bundle\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.306183 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.306153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-telemeter-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.306267 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.306203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-federate-client-tls\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.306502 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.306483    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.306535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.306515    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-secret-telemeter-client\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.313487 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.313461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4mh\" (UniqueName: \"kubernetes.io/projected/f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8-kube-api-access-6p4mh\") pod \"telemeter-client-784fff9fbd-9bwcq\" (UID: \"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8\") " pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.438547 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438510    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:21.438829 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438790    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="prometheus" containerID="cri-o://c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270" gracePeriod=600
Jul 29 21:19:21.438829 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438797    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy" containerID="cri-o://e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b" gracePeriod=600
Jul 29 21:19:21.439030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438855    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="thanos-sidecar" containerID="cri-o://a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3" gracePeriod=600
Jul 29 21:19:21.439030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438856    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-web" containerID="cri-o://0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88" gracePeriod=600
Jul 29 21:19:21.439030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438902    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-thanos" containerID="cri-o://80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a" gracePeriod=600
Jul 29 21:19:21.439030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.438860    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="config-reloader" containerID="cri-o://cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb" gracePeriod=600
Jul 29 21:19:21.475065 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.475019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"
Jul 29 21:19:21.618941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.618904    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-784fff9fbd-9bwcq"]
Jul 29 21:19:21.621884 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:19:21.621855    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88a5fe2_e7f9_46b2_94c4_d8f6accec4c8.slice/crio-4e070f1cec03e281ba9085622819639b3023499e88d58281dcb330952a6f4b2f WatchSource:0}: Error finding container 4e070f1cec03e281ba9085622819639b3023499e88d58281dcb330952a6f4b2f: Status 404 returned error can't find the container with id 4e070f1cec03e281ba9085622819639b3023499e88d58281dcb330952a6f4b2f
Jul 29 21:19:21.696615 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.696590    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:21.706814 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.706931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.706931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706851    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.706931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.706931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706944    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xl4k\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.706997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707079 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707074    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707109    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.707268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.707507 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707370    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.707952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708410    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708459    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708572    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.708596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708598    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls\") pod \"8b76f3db-0c86-4f89-a131-503dd2438fe1\" (UID: \"8b76f3db-0c86-4f89-a131-503dd2438fe1\") "
Jul 29 21:19:21.709061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708862    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.709061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708879    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.709061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.708894    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.709546 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.709267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:19:21.709664 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.709615    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.709834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.709794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out" (OuterVolumeSpecName: "config-out") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:19:21.709914 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.709893    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.710019 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.709906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:19:21.710348 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.710312    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.710422 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.710295    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.710768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.710726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config" (OuterVolumeSpecName: "config") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.710885 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.710781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.711291 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.711248    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k" (OuterVolumeSpecName: "kube-api-access-4xl4k") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "kube-api-access-4xl4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:19:21.711584 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.711556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.711662 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.711619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.711662 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.711647    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:19:21.711895 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.711874    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.712138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.712118    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.712991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.712973    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.722822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.722794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config" (OuterVolumeSpecName: "web-config") pod "8b76f3db-0c86-4f89-a131-503dd2438fe1" (UID: "8b76f3db-0c86-4f89-a131-503dd2438fe1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809686    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-metrics-client-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809716    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xl4k\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-kube-api-access-4xl4k\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809727    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809736    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-web-config\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809745    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809753    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809762    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809771    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-db\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.809775 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809782    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b76f3db-0c86-4f89-a131-503dd2438fe1-tls-assets\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809791    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809800    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809808    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-metrics-client-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809816    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809826    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b76f3db-0c86-4f89-a131-503dd2438fe1-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809834    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-grpc-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809842    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b76f3db-0c86-4f89-a131-503dd2438fe1-config-out\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.810161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.809849    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b76f3db-0c86-4f89-a131-503dd2438fe1-secret-kube-rbac-proxy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:19:21.981941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981910    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a" exitCode=0
Jul 29 21:19:21.981941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981934    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b" exitCode=0
Jul 29 21:19:21.981941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981940    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88" exitCode=0
Jul 29 21:19:21.981941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981946    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3" exitCode=0
Jul 29 21:19:21.981941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981951    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb" exitCode=0
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981956    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270" exitCode=0
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.981997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982025    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982075    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982140    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982199    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"}
Jul 29 21:19:21.982525 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.982281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b76f3db-0c86-4f89-a131-503dd2438fe1","Type":"ContainerDied","Data":"5cb22173a1597a91126c2ce9aca205358401a1068801c6305f38202211ba3fb5"}
Jul 29 21:19:21.985418 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.985339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq" event={"ID":"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8","Type":"ContainerStarted","Data":"4e070f1cec03e281ba9085622819639b3023499e88d58281dcb330952a6f4b2f"}
Jul 29 21:19:21.992712 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.992694    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:21.999591 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:21.999571    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.006568 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.006550    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.010026 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.009999    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:22.013776 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.013753    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:22.015675 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.015658    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.022841 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.022818    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.030083 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.030034    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.036709 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.036691    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.036968 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.036950    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.037006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.036980    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.037006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.036999    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.037365 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.037344    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.037456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037370    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.037456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037387    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.037690 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.037669    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.037730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.037730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037713    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.037951 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.037933    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.038002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037956    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.038002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.037975    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.038234 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.038214    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.038279 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038243    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.038279 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038262    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.038530 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.038514    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.038578 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038533    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.038578 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038547    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.038778 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:19:22.038762    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.038822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038782    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.038822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038796    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.039002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.038985    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.039035 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039003    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.039244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039223    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.039285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039245    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.039463 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039444    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.039504 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039464    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.039661 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039644    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.039726 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039661    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.039847 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039831    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.039887 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.039848    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.040076 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040059    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.040118 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040076    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.040258 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040243    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.040304 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040257    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.040458 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040438    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.040522 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040462    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.040700 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040683    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.040746 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040701    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.040889 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040871    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.040929 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.040890    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.041071 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041036    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.041110 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041071    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.041286 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041268    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.041347 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041287    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.041482 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041466    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.041532 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041483    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.041665 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041650    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.041665 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041664    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.041842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041825    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.041904 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.041844    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.042061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042033    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.042111 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042061    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.042252 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042234    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.042318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042254    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.042446 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042429    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.042487 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042445    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.042628 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042605    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.042673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042631    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.042821 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042804    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.042864 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042821    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.043005 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.042988    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.043064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043006    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.043201 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043185    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.043245 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043201    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.043406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043388    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.043481 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043410    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.043709 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043687    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.043748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043712    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.043966 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.044019 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.043966    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.044256 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044234    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.044302 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044257    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.044452 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044433    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.044492 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044454    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.044651 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044636    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.044692 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044652    2565 scope.go:117] "RemoveContainer" containerID="80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"
Jul 29 21:19:22.044834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044819    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a"} err="failed to get container status \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": rpc error: code = NotFound desc = could not find container \"80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a\": container with ID starting with 80bf2f79c69c6837494bfa0dde4b17132d53167bb19d410fba1bdf96e70d386a not found: ID does not exist"
Jul 29 21:19:22.044881 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.044834    2565 scope.go:117] "RemoveContainer" containerID="e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"
Jul 29 21:19:22.045030 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045011    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b"} err="failed to get container status \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": rpc error: code = NotFound desc = could not find container \"e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b\": container with ID starting with e76ffb0a0c55d542e5f82ef21aade5b6775cf1ff1232e043778a01279e49b28b not found: ID does not exist"
Jul 29 21:19:22.045133 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045032    2565 scope.go:117] "RemoveContainer" containerID="0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"
Jul 29 21:19:22.045335 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045318    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88"} err="failed to get container status \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": rpc error: code = NotFound desc = could not find container \"0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88\": container with ID starting with 0cf9426b00dd843d4f3c1c1f077630c4a8a3718bc2d26f9f752b75432b0ecf88 not found: ID does not exist"
Jul 29 21:19:22.045383 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045335    2565 scope.go:117] "RemoveContainer" containerID="a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"
Jul 29 21:19:22.045514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045500    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3"} err="failed to get container status \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": rpc error: code = NotFound desc = could not find container \"a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3\": container with ID starting with a19b88ee8e032f3f28d0668809409733273eace7fbd7ae8d0a5f0cbc34e2dfd3 not found: ID does not exist"
Jul 29 21:19:22.045558 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045516    2565 scope.go:117] "RemoveContainer" containerID="cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"
Jul 29 21:19:22.045724 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045706    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb"} err="failed to get container status \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": rpc error: code = NotFound desc = could not find container \"cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb\": container with ID starting with cab3bf07765be2593fd6b6c7b6ef05df48b6dbc24f851f9ccc7c2461e7bcd2eb not found: ID does not exist"
Jul 29 21:19:22.045783 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045725    2565 scope.go:117] "RemoveContainer" containerID="c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"
Jul 29 21:19:22.045923 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045907    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270"} err="failed to get container status \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": rpc error: code = NotFound desc = could not find container \"c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270\": container with ID starting with c3a2010fc9ad6f3a557473551322e1ad6f05d375d37dfd2c182d5add3560c270 not found: ID does not exist"
Jul 29 21:19:22.045969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.045922    2565 scope.go:117] "RemoveContainer" containerID="1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"
Jul 29 21:19:22.046127 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.046109    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76"} err="failed to get container status \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": rpc error: code = NotFound desc = could not find container \"1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76\": container with ID starting with 1b2edc5af1a846c4ea5cd13e11471ec27bf9bd25d2c094b9bbcf488b4690ec76 not found: ID does not exist"
Jul 29 21:19:22.053932 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.053903    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" path="/var/lib/kubelet/pods/8b76f3db-0c86-4f89-a131-503dd2438fe1/volumes"
Jul 29 21:19:22.054507 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054490    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:22.054752 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054740    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="thanos-sidecar"
Jul 29 21:19:22.054788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054754    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="thanos-sidecar"
Jul 29 21:19:22.054788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054764    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="prometheus"
Jul 29 21:19:22.054788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054769    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="prometheus"
Jul 29 21:19:22.054788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054781    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="config-reloader"
Jul 29 21:19:22.054788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054786    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="config-reloader"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054793    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-thanos"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-thanos"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054804    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-web"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-web"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="init-config-reloader"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054822    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="init-config-reloader"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054829    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054834    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054872    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="prometheus"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054879    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="thanos-sidecar"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054884    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-web"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054892    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="config-reloader"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054897    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy"
Jul 29 21:19:22.054944 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.054903    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b76f3db-0c86-4f89-a131-503dd2438fe1" containerName="kube-rbac-proxy-thanos"
Jul 29 21:19:22.060051 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.059995    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.062373 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.062352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 29 21:19:22.062517 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.062500    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 29 21:19:22.062580 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.062537    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 29 21:19:22.062824 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.062806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qsdqkjdipadp\""
Jul 29 21:19:22.063002 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.062986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 29 21:19:22.063118 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.063002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 29 21:19:22.063118 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.063016    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 29 21:19:22.063843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.063828    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 29 21:19:22.064062 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 29 21:19:22.064147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064071    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 29 21:19:22.064147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064106    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 29 21:19:22.064147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 29 21:19:22.064297 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064162    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 29 21:19:22.064487 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 29 21:19:22.065003 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.064917    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t6r44\""
Jul 29 21:19:22.072504 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.069215    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 29 21:19:22.072911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.072856    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 29 21:19:22.075842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.075818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:22.112679 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.112679 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.112884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.112884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.112884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112834    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.112884 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112918    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-kube-api-access-bjbbj\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.112998    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113194    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.113317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.113227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214091 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214010    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214091 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214191    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-kube-api-access-bjbbj\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.214784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214718    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.215333 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.214886    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.215473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.215435    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.215545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.215484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.215950 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.215922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.217747 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.217747 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217318    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.217747 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.217747 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217600    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.217747 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.218095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.217892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.218370 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.218348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.218666 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.218645    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.218910 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.218888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.219427 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.219407    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.219955 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.219935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.220135 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.220113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.220285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.220270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.226552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.226536    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbbj\" (UniqueName: \"kubernetes.io/projected/d9166be1-bbde-4752-b13e-0dd7cf04a5cb-kube-api-access-bjbbj\") pod \"prometheus-k8s-0\" (UID: \"d9166be1-bbde-4752-b13e-0dd7cf04a5cb\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.371192 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.371153    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:19:22.516882 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.516832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 29 21:19:22.519245 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:19:22.519216    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9166be1_bbde_4752_b13e_0dd7cf04a5cb.slice/crio-bf73a2d0830b740abb829bac5b670ec936d105a89396525c5936c50602c28b70 WatchSource:0}: Error finding container bf73a2d0830b740abb829bac5b670ec936d105a89396525c5936c50602c28b70: Status 404 returned error can't find the container with id bf73a2d0830b740abb829bac5b670ec936d105a89396525c5936c50602c28b70
Jul 29 21:19:22.992548 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.992501    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9166be1-bbde-4752-b13e-0dd7cf04a5cb" containerID="537bfcd227e0429c351309b3511036984a1d5fb4428054b32fba2e22870b7ed1" exitCode=0
Jul 29 21:19:22.992997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.992585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerDied","Data":"537bfcd227e0429c351309b3511036984a1d5fb4428054b32fba2e22870b7ed1"}
Jul 29 21:19:22.992997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:22.992663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"bf73a2d0830b740abb829bac5b670ec936d105a89396525c5936c50602c28b70"}
Jul 29 21:19:24.000335 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.000302    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq" event={"ID":"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8","Type":"ContainerStarted","Data":"5300084e97e494a10092c70bcf2cd4a2390b375819438e14964371183cb23e65"}
Jul 29 21:19:24.003993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.003968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"081cd0827686ce860380a738adade8b12bcd366eb5d6e031b1d08691236a92c1"}
Jul 29 21:19:24.004080 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.004063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"96888b5a6f644aa74c1169def429025fcbc892ed641bf5684cc0e78c0c00cfce"}
Jul 29 21:19:24.004123 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.004104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"fb21c79bf47d1e56f03dab5ddf2a80d27f8b00882ae6f1ccf4f4e53d69d3c890"}
Jul 29 21:19:24.004157 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.004142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"4a177adbd9215f4ce4a84b295f945d58911b369639baefbb71cbddc20531b55a"}
Jul 29 21:19:24.004191 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.004180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"bfa90553a1c19d8a5ce7669eb4453c18d8cad39770085ef2c5b8e8364a5c569a"}
Jul 29 21:19:24.004228 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.004219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d9166be1-bbde-4752-b13e-0dd7cf04a5cb","Type":"ContainerStarted","Data":"219b3852ef502e2400b605d102b40a9ac7823f4f705f6c3fbf52aee57a7b0c9a"}
Jul 29 21:19:24.036189 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:24.036119    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.016732362 podStartE2EDuration="2.036097954s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.365592ms" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:24.033001129 +0000 UTC m=+180.504580207" watchObservedRunningTime="2026-07-29 21:19:24.036097954 +0000 UTC m=+180.507677038"
Jul 29 21:19:25.010082 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:25.010027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq" event={"ID":"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8","Type":"ContainerStarted","Data":"1eab293a29832dceb3f52198cf9186351ad6e199d29410d476f02f19f3a3066b"}
Jul 29 21:19:25.010463 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:25.010107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq" event={"ID":"f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8","Type":"ContainerStarted","Data":"d11d90b930c6f7f26eb777ed53937a2c6f402adce6a6ca4d15554c8c66f0f6f9"}
Jul 29 21:19:25.041017 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:25.040960    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-784fff9fbd-9bwcq" podStartSLOduration=1.752731892 podStartE2EDuration="4.04094116s" totalImagesPullingTime="2.288209268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:25.039817328 +0000 UTC m=+181.511396404" watchObservedRunningTime="2026-07-29 21:19:25.04094116 +0000 UTC m=+181.512520240"
Jul 29 21:19:27.372128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:19:27.372090    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:20:22.371667 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:20:22.371625    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:20:22.387197 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:20:22.387167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:20:23.246498 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:20:23.246469    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 29 21:21:23.947642 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:21:23.947612    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 29 21:22:09.747256 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.747214    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"]
Jul 29 21:22:09.750484 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.750466    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.752855 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.752834    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 29 21:22:09.753589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.753573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-7ndh9\""
Jul 29 21:22:09.753589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.753585    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 29 21:22:09.764193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.764167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"]
Jul 29 21:22:09.806370 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.806341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.806370 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.806371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknzq\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-kube-api-access-kknzq\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.907722 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.907685    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.907860 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.907743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknzq\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-kube-api-access-kknzq\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.917014 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.916977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:09.917233 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:09.917204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknzq\" (UniqueName: \"kubernetes.io/projected/3adc099d-0846-40bf-bee2-068c343b21c7-kube-api-access-kknzq\") pod \"cert-manager-cainjector-559cdd8c87-45k7m\" (UID: \"3adc099d-0846-40bf-bee2-068c343b21c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:10.073224 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:10.073120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"
Jul 29 21:22:10.203918 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:10.203892    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-45k7m"]
Jul 29 21:22:10.206756 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:22:10.206734    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3adc099d_0846_40bf_bee2_068c343b21c7.slice/crio-83243a8f64f2e010ca1827a76e60860012d7cd5856f2b2e150421867d09d22e5 WatchSource:0}: Error finding container 83243a8f64f2e010ca1827a76e60860012d7cd5856f2b2e150421867d09d22e5: Status 404 returned error can't find the container with id 83243a8f64f2e010ca1827a76e60860012d7cd5856f2b2e150421867d09d22e5
Jul 29 21:22:10.208977 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:10.208960    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:22:10.625992 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:10.625957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m" event={"ID":"3adc099d-0846-40bf-bee2-068c343b21c7","Type":"ContainerStarted","Data":"83243a8f64f2e010ca1827a76e60860012d7cd5856f2b2e150421867d09d22e5"}
Jul 29 21:22:14.644461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:14.644420    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m" event={"ID":"3adc099d-0846-40bf-bee2-068c343b21c7","Type":"ContainerStarted","Data":"16a4fee0c7687e4a2f297ff2acd132f5f4ffa06aa8f5c337f2a79202141689cc"}
Jul 29 21:22:14.662481 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:14.662427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-45k7m" podStartSLOduration=2.101580245 podStartE2EDuration="5.662409721s" totalImagesPullingTime="3.560829476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:14.661358582 +0000 UTC m=+351.132937658" watchObservedRunningTime="2026-07-29 21:22:14.662409721 +0000 UTC m=+351.133988794"
Jul 29 21:22:43.158425 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.158339    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"]
Jul 29 21:22:43.164987 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.164963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.167354 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.167327    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 29 21:22:43.167514 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.167494    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 29 21:22:43.167586 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.167511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 29 21:22:43.167586 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.167496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sk4g6\""
Jul 29 21:22:43.167667 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.167585    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 29 21:22:43.177283 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.177257    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"]
Jul 29 21:22:43.291634 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.291598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.291634 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.291637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.291832 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.291678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.291832 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.291696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n684c\" (UniqueName: \"kubernetes.io/projected/107ebb15-70e2-4433-8ac8-c388bfa2f010-kube-api-access-n684c\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.291832 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.291734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393232 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393286    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393302    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n684c\" (UniqueName: \"kubernetes.io/projected/107ebb15-70e2-4433-8ac8-c388bfa2f010-kube-api-access-n684c\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393548 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393455    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.393719 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.393698    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.395543 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.395523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/107ebb15-70e2-4433-8ac8-c388bfa2f010-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.395677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.395660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.395860 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.395842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/107ebb15-70e2-4433-8ac8-c388bfa2f010-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.401740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.401719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n684c\" (UniqueName: \"kubernetes.io/projected/107ebb15-70e2-4433-8ac8-c388bfa2f010-kube-api-access-n684c\") pod \"opendatahub-operator-controller-manager-6b55745745-pwbq8\" (UID: \"107ebb15-70e2-4433-8ac8-c388bfa2f010\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.475951 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.475919    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:43.614677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.614653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"]
Jul 29 21:22:43.617035 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:22:43.617005    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107ebb15_70e2_4433_8ac8_c388bfa2f010.slice/crio-a96e3693c7e65522cd461424e3b700081adccf237135e28290c8a5a7ad6e6e44 WatchSource:0}: Error finding container a96e3693c7e65522cd461424e3b700081adccf237135e28290c8a5a7ad6e6e44: Status 404 returned error can't find the container with id a96e3693c7e65522cd461424e3b700081adccf237135e28290c8a5a7ad6e6e44
Jul 29 21:22:43.756075 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:43.755958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8" event={"ID":"107ebb15-70e2-4433-8ac8-c388bfa2f010","Type":"ContainerStarted","Data":"a96e3693c7e65522cd461424e3b700081adccf237135e28290c8a5a7ad6e6e44"}
Jul 29 21:22:47.775319 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:47.775273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8" event={"ID":"107ebb15-70e2-4433-8ac8-c388bfa2f010","Type":"ContainerStarted","Data":"aafc9c48cbad767708470918a7650cac6e55bde8a8463ff315f31f4af20aec03"}
Jul 29 21:22:48.781521 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:48.781488    2565 generic.go:356] "Generic (PLEG): container finished" podID="107ebb15-70e2-4433-8ac8-c388bfa2f010" containerID="aafc9c48cbad767708470918a7650cac6e55bde8a8463ff315f31f4af20aec03" exitCode=0
Jul 29 21:22:48.781879 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:48.781542    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8" event={"ID":"107ebb15-70e2-4433-8ac8-c388bfa2f010","Type":"ContainerDied","Data":"aafc9c48cbad767708470918a7650cac6e55bde8a8463ff315f31f4af20aec03"}
Jul 29 21:22:49.787954 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:49.787917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8" event={"ID":"107ebb15-70e2-4433-8ac8-c388bfa2f010","Type":"ContainerStarted","Data":"d8907f5cbf91d0c2d4f72d51564c1ba4e9ae08defe63857b2b4b645818048cc3"}
Jul 29 21:22:49.788355 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:49.788016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:22:49.808400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:22:49.808351    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8" podStartSLOduration=2.200204963 podStartE2EDuration="6.808336024s" totalImagesPullingTime="3.581310055s" totalInitContainerRuntime="1.026821006s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:49.807195762 +0000 UTC m=+386.278774844" watchObservedRunningTime="2026-07-29 21:22:49.808336024 +0000 UTC m=+386.279915100"
Jul 29 21:23:00.795019 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:00.794986    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-pwbq8"
Jul 29 21:23:12.039008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.038969    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"]
Jul 29 21:23:12.052845 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.052816    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.056009 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.055986    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 29 21:23:12.056845 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.056826    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2k4ht\""
Jul 29 21:23:12.057081 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.057066    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 29 21:23:12.057171 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.057129    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 29 21:23:12.057266 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.057231    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 29 21:23:12.057390 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.057296    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 29 21:23:12.062637 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.062614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"]
Jul 29 21:23:12.143992 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.143946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48zgl\" (UniqueName: \"kubernetes.io/projected/c1f8c9a5-132d-499e-9301-3db3628bcf2a-kube-api-access-48zgl\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.144689 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.144659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.144961 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.144941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1f8c9a5-132d-499e-9301-3db3628bcf2a-manager-config\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.145618 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.145598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.247073 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.247024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1f8c9a5-132d-499e-9301-3db3628bcf2a-manager-config\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.247279 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.247102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.247279 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.247140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48zgl\" (UniqueName: \"kubernetes.io/projected/c1f8c9a5-132d-499e-9301-3db3628bcf2a-kube-api-access-48zgl\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.247279 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.247161    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.247704 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.247676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c1f8c9a5-132d-499e-9301-3db3628bcf2a-manager-config\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.249653 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.249622    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.250102 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.250030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1f8c9a5-132d-499e-9301-3db3628bcf2a-cert\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.261689 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.261656    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48zgl\" (UniqueName: \"kubernetes.io/projected/c1f8c9a5-132d-499e-9301-3db3628bcf2a-kube-api-access-48zgl\") pod \"lws-controller-manager-5999ff5f6f-bg8lv\" (UID: \"c1f8c9a5-132d-499e-9301-3db3628bcf2a\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.364532 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.364436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:12.509285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.509198    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"]
Jul 29 21:23:12.512222 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:23:12.512188    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f8c9a5_132d_499e_9301_3db3628bcf2a.slice/crio-2c49ecaa887500d8f794c446b73a5de7f60c19c905bb71df87ab079f3b899b4f WatchSource:0}: Error finding container 2c49ecaa887500d8f794c446b73a5de7f60c19c905bb71df87ab079f3b899b4f: Status 404 returned error can't find the container with id 2c49ecaa887500d8f794c446b73a5de7f60c19c905bb71df87ab079f3b899b4f
Jul 29 21:23:12.887894 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:12.887823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv" event={"ID":"c1f8c9a5-132d-499e-9301-3db3628bcf2a","Type":"ContainerStarted","Data":"2c49ecaa887500d8f794c446b73a5de7f60c19c905bb71df87ab079f3b899b4f"}
Jul 29 21:23:14.690563 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.690528    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"]
Jul 29 21:23:14.694380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.694353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.699188 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.699160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 29 21:23:14.717001 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.705703    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"]
Jul 29 21:23:14.717001 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.709120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7tj26\""
Jul 29 21:23:14.723522 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.723478    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"]
Jul 29 21:23:14.723680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.723609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.727085 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.726916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 29 21:23:14.727373 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.727351    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-kxrnn\""
Jul 29 21:23:14.729799 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.729772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"]
Jul 29 21:23:14.774150 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.774110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhmn5\" (UniqueName: \"kubernetes.io/projected/69d45d59-fdb0-44a7-98a6-a41677f79a0b-kube-api-access-zhmn5\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.774150 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.774154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.874641 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.874601    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhmn5\" (UniqueName: \"kubernetes.io/projected/69d45d59-fdb0-44a7-98a6-a41677f79a0b-kube-api-access-zhmn5\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.874798 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.874650    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.874798 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.874709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.874798 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.874751    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7824\" (UniqueName: \"kubernetes.io/projected/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-kube-api-access-p7824\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.874910 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:23:14.874825    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 29 21:23:14.874910 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:23:14.874888    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert podName:69d45d59-fdb0-44a7-98a6-a41677f79a0b nodeName:}" failed. No retries permitted until 2026-07-29 21:23:15.374872874 +0000 UTC m=+411.846451929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert") pod "kserve-controller-manager-d6f4b65b8-z6hq8" (UID: "69d45d59-fdb0-44a7-98a6-a41677f79a0b") : secret "kserve-webhook-server-cert" not found
Jul 29 21:23:14.910665 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.910625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhmn5\" (UniqueName: \"kubernetes.io/projected/69d45d59-fdb0-44a7-98a6-a41677f79a0b-kube-api-access-zhmn5\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:14.976240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.976125    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.976240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.976207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7824\" (UniqueName: \"kubernetes.io/projected/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-kube-api-access-p7824\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.979417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.979385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:14.990802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:14.990778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7824\" (UniqueName: \"kubernetes.io/projected/d6be8fa7-28a8-45c4-8847-d94ee8e48c81-kube-api-access-p7824\") pod \"llmisvc-controller-manager-777c6bd7d5-njl7k\" (UID: \"d6be8fa7-28a8-45c4-8847-d94ee8e48c81\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:15.037529 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.037498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:15.209013 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.208925    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"]
Jul 29 21:23:15.217494 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:23:15.217460    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6be8fa7_28a8_45c4_8847_d94ee8e48c81.slice/crio-ffd5519ee879d4499bf38e7f54d2670708deab3f600ce123f9c229d0390ccee1 WatchSource:0}: Error finding container ffd5519ee879d4499bf38e7f54d2670708deab3f600ce123f9c229d0390ccee1: Status 404 returned error can't find the container with id ffd5519ee879d4499bf38e7f54d2670708deab3f600ce123f9c229d0390ccee1
Jul 29 21:23:15.380610 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.380564    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:15.380817 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:23:15.380789    2565 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 29 21:23:15.380886 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:23:15.380858    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert podName:69d45d59-fdb0-44a7-98a6-a41677f79a0b nodeName:}" failed. No retries permitted until 2026-07-29 21:23:16.380838839 +0000 UTC m=+412.852417897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert") pod "kserve-controller-manager-d6f4b65b8-z6hq8" (UID: "69d45d59-fdb0-44a7-98a6-a41677f79a0b") : secret "kserve-webhook-server-cert" not found
Jul 29 21:23:15.903016 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.902981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv" event={"ID":"c1f8c9a5-132d-499e-9301-3db3628bcf2a","Type":"ContainerStarted","Data":"ad3a1217fe7ab1db0017fa386b2be6824bdd0a2bf0b42f5a611fb22385094534"}
Jul 29 21:23:15.903438 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.903080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:15.904060 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.904018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k" event={"ID":"d6be8fa7-28a8-45c4-8847-d94ee8e48c81","Type":"ContainerStarted","Data":"ffd5519ee879d4499bf38e7f54d2670708deab3f600ce123f9c229d0390ccee1"}
Jul 29 21:23:15.943584 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:15.943497    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv" podStartSLOduration=1.523463333 podStartE2EDuration="3.943471613s" totalImagesPullingTime="2.42000828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:15.942562163 +0000 UTC m=+412.414141239" watchObservedRunningTime="2026-07-29 21:23:15.943471613 +0000 UTC m=+412.415050689"
Jul 29 21:23:16.274993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.274941    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"]
Jul 29 21:23:16.279107 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.279084    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.281810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.281786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 29 21:23:16.281927 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.281794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-m4gp9\""
Jul 29 21:23:16.282663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.282640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 29 21:23:16.282954 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.282933    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 29 21:23:16.295667 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.295640    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"]
Jul 29 21:23:16.391198 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391354 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391354 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391354 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391354 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrkz\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-kube-api-access-dsrkz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:16.391520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.391520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.391515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.393848 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.393817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/69d45d59-fdb0-44a7-98a6-a41677f79a0b-cert\") pod \"kserve-controller-manager-d6f4b65b8-z6hq8\" (UID: \"69d45d59-fdb0-44a7-98a6-a41677f79a0b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:16.492183 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492183 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrkz\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-kube-api-access-dsrkz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492424    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492773 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492867 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492946 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.492998 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.492969    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.493215 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.493193    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.494827 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.494801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.495170 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.495153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.507417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.507390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:16.508228 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.508204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.509718 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.509677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrkz\" (UniqueName: \"kubernetes.io/projected/c686c0c7-6a0c-4ecd-93ae-c9db3a21b241-kube-api-access-dsrkz\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n\" (UID: \"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.592561 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.592471    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:16.723406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.723373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"]
Jul 29 21:23:16.725731 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:23:16.725703    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d45d59_fdb0_44a7_98a6_a41677f79a0b.slice/crio-f034dc20df1181693e4f5149536d6151cd9f1749e6ee893cbaec26db5dde8dac WatchSource:0}: Error finding container f034dc20df1181693e4f5149536d6151cd9f1749e6ee893cbaec26db5dde8dac: Status 404 returned error can't find the container with id f034dc20df1181693e4f5149536d6151cd9f1749e6ee893cbaec26db5dde8dac
Jul 29 21:23:16.765078 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.765023    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"]
Jul 29 21:23:16.769595 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:23:16.769569    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc686c0c7_6a0c_4ecd_93ae_c9db3a21b241.slice/crio-cb88cafcc9b8a5945e4dde3c654e0a88b4c16e122d94e82986cde7251ae3f209 WatchSource:0}: Error finding container cb88cafcc9b8a5945e4dde3c654e0a88b4c16e122d94e82986cde7251ae3f209: Status 404 returned error can't find the container with id cb88cafcc9b8a5945e4dde3c654e0a88b4c16e122d94e82986cde7251ae3f209
Jul 29 21:23:16.909676 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.909582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n" event={"ID":"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241","Type":"ContainerStarted","Data":"cb88cafcc9b8a5945e4dde3c654e0a88b4c16e122d94e82986cde7251ae3f209"}
Jul 29 21:23:16.910759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:16.910730    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8" event={"ID":"69d45d59-fdb0-44a7-98a6-a41677f79a0b","Type":"ContainerStarted","Data":"f034dc20df1181693e4f5149536d6151cd9f1749e6ee893cbaec26db5dde8dac"}
Jul 29 21:23:18.923138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:18.923068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k" event={"ID":"d6be8fa7-28a8-45c4-8847-d94ee8e48c81","Type":"ContainerStarted","Data":"cc1ba94fd514be62a2d8b129b503a44ae15e869b023894212c97971284d040c2"}
Jul 29 21:23:18.923603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:18.923184    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:18.945563 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:18.944916    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k" podStartSLOduration=2.012809833 podStartE2EDuration="4.944893379s" totalImagesPullingTime="2.932083546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:18.94201166 +0000 UTC m=+415.413590758" watchObservedRunningTime="2026-07-29 21:23:18.944893379 +0000 UTC m=+415.416472457"
Jul 29 21:23:20.339394 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.339356    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:23:20.339680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.339429    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:23:20.339680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.339456    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:23:20.933099 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.933032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n" event={"ID":"c686c0c7-6a0c-4ecd-93ae-c9db3a21b241","Type":"ContainerStarted","Data":"156ec8972894b9fd89f794a4bcb84b5f1dd7a8b0c122089cb54d6849bea15be9"}
Jul 29 21:23:20.934473 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.934451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8" event={"ID":"69d45d59-fdb0-44a7-98a6-a41677f79a0b","Type":"ContainerStarted","Data":"d463964d479f48524e66af2879f50bb2f600b8f31b330199afaea13e8cc25cb6"}
Jul 29 21:23:20.934563 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.934504    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:23:20.958159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.958111    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n" podStartSLOduration=1.390838404 podStartE2EDuration="4.958095851s" totalImagesPullingTime="3.567257447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:20.956141956 +0000 UTC m=+417.427721035" watchObservedRunningTime="2026-07-29 21:23:20.958095851 +0000 UTC m=+417.429674927"
Jul 29 21:23:20.979597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:20.979544    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8" podStartSLOduration=3.422970342 podStartE2EDuration="6.979529881s" totalImagesPullingTime="3.556559539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:20.978155566 +0000 UTC m=+417.449734641" watchObservedRunningTime="2026-07-29 21:23:20.979529881 +0000 UTC m=+417.451108957"
Jul 29 21:23:21.592945 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:21.592901    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:21.598332 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:21.598302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:21.942061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:21.942007    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:21.943020 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:21.943002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n"
Jul 29 21:23:26.913454 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:26.913421    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-bg8lv"
Jul 29 21:23:49.930391 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:49.930322    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-njl7k"
Jul 29 21:23:51.947072 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:23:51.947024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-z6hq8"
Jul 29 21:24:34.753457 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.753418    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6945bf6885-kgn59"]
Jul 29 21:24:34.756791 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.756770    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.759200 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.759175    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 29 21:24:34.759200 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.759176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 29 21:24:34.760186 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.760168    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 29 21:24:34.760285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.760218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 29 21:24:34.760346 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.760219    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 29 21:24:34.760429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.760411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qrndx\""
Jul 29 21:24:34.770698 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.770667    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 29 21:24:34.770848 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.770817    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6945bf6885-kgn59"]
Jul 29 21:24:34.875121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-oauth-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-oauth-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875393 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fft27\" (UniqueName: \"kubernetes.io/projected/06c71a14-8599-4975-9dae-55dbf5feb873-kube-api-access-fft27\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875393 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875281    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-console-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875393 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-trusted-ca-bundle\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.875534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.875438    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-service-ca\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976402 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-console-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-trusted-ca-bundle\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976492    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-service-ca\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-oauth-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976603 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-oauth-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.976872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.976761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fft27\" (UniqueName: \"kubernetes.io/projected/06c71a14-8599-4975-9dae-55dbf5feb873-kube-api-access-fft27\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.977285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.977258    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-console-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.977285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.977270    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-service-ca\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.977462 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.977358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-oauth-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.977462 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.977375    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06c71a14-8599-4975-9dae-55dbf5feb873-trusted-ca-bundle\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.978969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.978944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-serving-cert\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.978969 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.978961    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/06c71a14-8599-4975-9dae-55dbf5feb873-console-oauth-config\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:34.986459 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:34.986439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fft27\" (UniqueName: \"kubernetes.io/projected/06c71a14-8599-4975-9dae-55dbf5feb873-kube-api-access-fft27\") pod \"console-6945bf6885-kgn59\" (UID: \"06c71a14-8599-4975-9dae-55dbf5feb873\") " pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:35.072915 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:35.072825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:35.210077 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:35.210031    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6945bf6885-kgn59"]
Jul 29 21:24:35.212961 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:24:35.212937    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c71a14_8599_4975_9dae_55dbf5feb873.slice/crio-8e7aaef8b58ec2c595831c8536489e9025d8402bf7d48022169e50709f007c14 WatchSource:0}: Error finding container 8e7aaef8b58ec2c595831c8536489e9025d8402bf7d48022169e50709f007c14: Status 404 returned error can't find the container with id 8e7aaef8b58ec2c595831c8536489e9025d8402bf7d48022169e50709f007c14
Jul 29 21:24:35.253111 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:35.253081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6945bf6885-kgn59" event={"ID":"06c71a14-8599-4975-9dae-55dbf5feb873","Type":"ContainerStarted","Data":"8e7aaef8b58ec2c595831c8536489e9025d8402bf7d48022169e50709f007c14"}
Jul 29 21:24:36.259223 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:36.259182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6945bf6885-kgn59" event={"ID":"06c71a14-8599-4975-9dae-55dbf5feb873","Type":"ContainerStarted","Data":"0692478a7afbbf1836872c0c7eb1870b35cd745bca004898a83980a2af377610"}
Jul 29 21:24:36.278243 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:36.278186    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6945bf6885-kgn59" podStartSLOduration=2.278168356 podStartE2EDuration="2.278168356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:24:36.276436784 +0000 UTC m=+492.748015860" watchObservedRunningTime="2026-07-29 21:24:36.278168356 +0000 UTC m=+492.749747432"
Jul 29 21:24:45.073178 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:45.073136    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:45.073581 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:45.073209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:45.077953 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:45.077926    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:24:45.307722 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:24:45.307689    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6945bf6885-kgn59"
Jul 29 21:25:16.907835 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.907742    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"]
Jul 29 21:25:16.911449 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.911427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:16.915497 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.915478    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 21:25:16.915583 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.915482    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 21:25:16.916273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.916257    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-nm827\""
Jul 29 21:25:16.934089 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.934061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"]
Jul 29 21:25:16.945957 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.945933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:16.946097 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:16.946014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5ml\" (UniqueName: \"kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.046958 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.046919    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.047168 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.047006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5ml\" (UniqueName: \"kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.047294 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.047273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.059834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.059812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5ml\" (UniqueName: \"kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.221980 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.221940    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:17.360953 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.360820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"]
Jul 29 21:25:17.364226 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:25:17.364197    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a439a92_26a4_479c_a66a_af90e82e74b6.slice/crio-610f99bd7f10cf93eb63f9a51bd25c9253136a8427166a53f4a9cf270200a440 WatchSource:0}: Error finding container 610f99bd7f10cf93eb63f9a51bd25c9253136a8427166a53f4a9cf270200a440: Status 404 returned error can't find the container with id 610f99bd7f10cf93eb63f9a51bd25c9253136a8427166a53f4a9cf270200a440
Jul 29 21:25:17.448888 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.448852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv" event={"ID":"2a439a92-26a4-479c-a66a-af90e82e74b6","Type":"ContainerStarted","Data":"610f99bd7f10cf93eb63f9a51bd25c9253136a8427166a53f4a9cf270200a440"}
Jul 29 21:25:17.661147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.661057    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"]
Jul 29 21:25:17.668225 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.668186    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"]
Jul 29 21:25:17.689835 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.689804    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:17.693189 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.693165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.716618 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.716590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:17.754705 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.754675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.754850 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.754768    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfqfd\" (UniqueName: \"kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.855794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.855752    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfqfd\" (UniqueName: \"kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.855966 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.855844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.856205 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.856187    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:17.870455 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:17.870427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfqfd\" (UniqueName: \"kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dn9bn\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:18.005291 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:18.005253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:18.580289 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:18.580266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:18.624189 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:25:18.624138    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf97909_e604_4c0f_9349_b75c9b5ac300.slice/crio-14907a3b2428e2fe9336c5be11ccfadc91a657b7b1a28f28f02851c47d4877c6 WatchSource:0}: Error finding container 14907a3b2428e2fe9336c5be11ccfadc91a657b7b1a28f28f02851c47d4877c6: Status 404 returned error can't find the container with id 14907a3b2428e2fe9336c5be11ccfadc91a657b7b1a28f28f02851c47d4877c6
Jul 29 21:25:19.462567 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:19.462515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" event={"ID":"6cf97909-e604-4c0f-9349-b75c9b5ac300","Type":"ContainerStarted","Data":"14907a3b2428e2fe9336c5be11ccfadc91a657b7b1a28f28f02851c47d4877c6"}
Jul 29 21:25:22.479337 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.479284    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" containerName="manager" containerID="cri-o://9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee" gracePeriod=2
Jul 29 21:25:22.480926 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.480895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" event={"ID":"6cf97909-e604-4c0f-9349-b75c9b5ac300","Type":"ContainerStarted","Data":"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"}
Jul 29 21:25:22.481058 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.480993    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:22.504852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.504802    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" podStartSLOduration=2.210422106 podStartE2EDuration="5.504788325s" totalImagesPullingTime="3.294366219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:25:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:25:22.502348662 +0000 UTC m=+538.973927770" watchObservedRunningTime="2026-07-29 21:25:22.504788325 +0000 UTC m=+538.976367401"
Jul 29 21:25:22.714649 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.714626    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:22.716696 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:22.716668    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:22.805196 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.805119    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume\") pod \"2a439a92-26a4-479c-a66a-af90e82e74b6\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") "
Jul 29 21:25:22.805196 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.805179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w5ml\" (UniqueName: \"kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml\") pod \"2a439a92-26a4-479c-a66a-af90e82e74b6\" (UID: \"2a439a92-26a4-479c-a66a-af90e82e74b6\") "
Jul 29 21:25:22.805436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.805408    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2a439a92-26a4-479c-a66a-af90e82e74b6" (UID: "2a439a92-26a4-479c-a66a-af90e82e74b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:22.807411 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.807389    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml" (OuterVolumeSpecName: "kube-api-access-4w5ml") pod "2a439a92-26a4-479c-a66a-af90e82e74b6" (UID: "2a439a92-26a4-479c-a66a-af90e82e74b6"). InnerVolumeSpecName "kube-api-access-4w5ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:25:22.906126 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.906094    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2a439a92-26a4-479c-a66a-af90e82e74b6-extensions-socket-volume\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:22.906126 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:22.906120    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5ml\" (UniqueName: \"kubernetes.io/projected/2a439a92-26a4-479c-a66a-af90e82e74b6-kube-api-access-4w5ml\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:23.489276 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:23.489236    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a439a92-26a4-479c-a66a-af90e82e74b6" containerID="9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee" exitCode=2
Jul 29 21:25:23.489686 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:23.489298    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:23.489686 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:23.489329    2565 scope.go:117] "RemoveContainer" containerID="9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee"
Jul 29 21:25:23.491548 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:23.491516    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:23.498257 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:23.498236    2565 scope.go:117] "RemoveContainer" containerID="9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee"
Jul 29 21:25:23.498536 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:23.498518    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee\": container with ID starting with 9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee not found: ID does not exist" containerID="9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee"
Jul 29 21:25:23.498594 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:23.498546    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee"} err="failed to get container status \"9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee\": rpc error: code = NotFound desc = could not find container \"9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee\": container with ID starting with 9d10a44fa78c1b149bceb10a09bc744168ec353025b197207d494627a61708ee not found: ID does not exist"
Jul 29 21:25:23.499945 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:23.499925    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:24.052868 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:24.052833    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-m7mgv\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-m7mgv"
Jul 29 21:25:24.054240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:24.054214    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" path="/var/lib/kubelet/pods/2a439a92-26a4-479c-a66a-af90e82e74b6/volumes"
Jul 29 21:25:33.492037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:33.492004    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:46.057021 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.056989    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:46.057545 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.057182    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" podUID="6cf97909-e604-4c0f-9349-b75c9b5ac300" containerName="manager" containerID="cri-o://84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d" gracePeriod=10
Jul 29 21:25:46.307420 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.307365    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:46.320539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.320506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume\") pod \"6cf97909-e604-4c0f-9349-b75c9b5ac300\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") "
Jul 29 21:25:46.320708 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.320615    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfqfd\" (UniqueName: \"kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd\") pod \"6cf97909-e604-4c0f-9349-b75c9b5ac300\" (UID: \"6cf97909-e604-4c0f-9349-b75c9b5ac300\") "
Jul 29 21:25:46.320907 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.320881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6cf97909-e604-4c0f-9349-b75c9b5ac300" (UID: "6cf97909-e604-4c0f-9349-b75c9b5ac300"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:25:46.322500 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.322472    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd" (OuterVolumeSpecName: "kube-api-access-pfqfd") pod "6cf97909-e604-4c0f-9349-b75c9b5ac300" (UID: "6cf97909-e604-4c0f-9349-b75c9b5ac300"). InnerVolumeSpecName "kube-api-access-pfqfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:25:46.421917 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.421882    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6cf97909-e604-4c0f-9349-b75c9b5ac300-extensions-socket-volume\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:46.421917 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.421911    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfqfd\" (UniqueName: \"kubernetes.io/projected/6cf97909-e604-4c0f-9349-b75c9b5ac300-kube-api-access-pfqfd\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:25:46.594538 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.594441    2565 generic.go:356] "Generic (PLEG): container finished" podID="6cf97909-e604-4c0f-9349-b75c9b5ac300" containerID="84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d" exitCode=0
Jul 29 21:25:46.594538 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.594488    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"
Jul 29 21:25:46.594716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.594531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" event={"ID":"6cf97909-e604-4c0f-9349-b75c9b5ac300","Type":"ContainerDied","Data":"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"}
Jul 29 21:25:46.594716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.594588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn" event={"ID":"6cf97909-e604-4c0f-9349-b75c9b5ac300","Type":"ContainerDied","Data":"14907a3b2428e2fe9336c5be11ccfadc91a657b7b1a28f28f02851c47d4877c6"}
Jul 29 21:25:46.594716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.594617    2565 scope.go:117] "RemoveContainer" containerID="84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"
Jul 29 21:25:46.604094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.604079    2565 scope.go:117] "RemoveContainer" containerID="84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"
Jul 29 21:25:46.604379 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:25:46.604355    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d\": container with ID starting with 84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d not found: ID does not exist" containerID="84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"
Jul 29 21:25:46.604470 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.604386    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d"} err="failed to get container status \"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d\": rpc error: code = NotFound desc = could not find container \"84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d\": container with ID starting with 84086998d3651a111a10595678780c1cc8f9be6d360edc17229b645e4837de1d not found: ID does not exist"
Jul 29 21:25:46.620936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.620905    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:46.625587 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:46.625562    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dn9bn"]
Jul 29 21:25:48.054834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:48.054800    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cf97909-e604-4c0f-9349-b75c9b5ac300" path="/var/lib/kubelet/pods/6cf97909-e604-4c0f-9349-b75c9b5ac300/volumes"
Jul 29 21:25:50.054263 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054222    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"]
Jul 29 21:25:50.054617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054555    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cf97909-e604-4c0f-9349-b75c9b5ac300" containerName="manager"
Jul 29 21:25:50.054617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054566    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cf97909-e604-4c0f-9349-b75c9b5ac300" containerName="manager"
Jul 29 21:25:50.054617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054575    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" containerName="manager"
Jul 29 21:25:50.054617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054580    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" containerName="manager"
Jul 29 21:25:50.054740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054637    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cf97909-e604-4c0f-9349-b75c9b5ac300" containerName="manager"
Jul 29 21:25:50.054740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.054649    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a439a92-26a4-479c-a66a-af90e82e74b6" containerName="manager"
Jul 29 21:25:50.056843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.056829    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.059142 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.059121    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ldr5c\""
Jul 29 21:25:50.067770 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.067738    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"]
Jul 29 21:25:50.154443 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ttx7\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-kube-api-access-4ttx7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154556    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154878 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc54791-82ee-46cc-bc56-3497049bfef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.154878 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.154712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256100 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256100 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ttx7\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-kube-api-access-4ttx7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256462    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc54791-82ee-46cc-bc56-3497049bfef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256750 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.256750 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.256697    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.257094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.257073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc54791-82ee-46cc-bc56-3497049bfef7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.258516 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.258484    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.258605 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.258562    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.264955 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.264926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.265247 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.265219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ttx7\" (UniqueName: \"kubernetes.io/projected/3bc54791-82ee-46cc-bc56-3497049bfef7-kube-api-access-4ttx7\") pod \"maas-default-gateway-openshift-default-78bb78ff78-mjv7r\" (UID: \"3bc54791-82ee-46cc-bc56-3497049bfef7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.367577 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.367488    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:50.503126 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.503090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"]
Jul 29 21:25:50.506275 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:25:50.506245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc54791_82ee_46cc_bc56_3497049bfef7.slice/crio-ec6eb2e53b44500c478f3809fb9a56185cd03c6b9d7c5a2215922c45918c109e WatchSource:0}: Error finding container ec6eb2e53b44500c478f3809fb9a56185cd03c6b9d7c5a2215922c45918c109e: Status 404 returned error can't find the container with id ec6eb2e53b44500c478f3809fb9a56185cd03c6b9d7c5a2215922c45918c109e
Jul 29 21:25:50.508273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.508241    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:25:50.508374 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.508316    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:25:50.508374 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.508364    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:25:50.616497 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.616458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" event={"ID":"3bc54791-82ee-46cc-bc56-3497049bfef7","Type":"ContainerStarted","Data":"cd640e11cd982c20a8e27c7968f5b6250809deef67ca91e42552b8778b8844be"}
Jul 29 21:25:50.616677 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.616508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" event={"ID":"3bc54791-82ee-46cc-bc56-3497049bfef7","Type":"ContainerStarted","Data":"ec6eb2e53b44500c478f3809fb9a56185cd03c6b9d7c5a2215922c45918c109e"}
Jul 29 21:25:50.640413 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:50.640293    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podStartSLOduration=0.640272828 podStartE2EDuration="640.272828ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:25:50.63675537 +0000 UTC m=+567.108334443" watchObservedRunningTime="2026-07-29 21:25:50.640272828 +0000 UTC m=+567.111851904"
Jul 29 21:25:51.368098 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:51.368031    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:51.369553 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:51.369524    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mjv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 29 21:25:51.369666 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:51.369583    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podUID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 29 21:25:52.371520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:52.371491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:52.625670 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:52.625590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:25:52.626421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:25:52.626400    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:26:05.107091 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.107034    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:05.110597 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.110578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.112899 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.112879    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 29 21:26:05.113826 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.113808    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ztd8p\""
Jul 29 21:26:05.113873 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.113824    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 29 21:26:05.113873 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.113855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 29 21:26:05.119096 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.119076    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:05.182711 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.182673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x295\" (UniqueName: \"kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.182873 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.182744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.200980 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.200948    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:05.283591 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.283555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x295\" (UniqueName: \"kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.283759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.283605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.284235 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.284217    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.293542 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.293516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x295\" (UniqueName: \"kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295\") pod \"limitador-limitador-64c94fd64f-n7pxv\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.421920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.421872    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:05.553456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.553356    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:05.555815 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:26:05.555791    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0ed0ef_4ba9_488d_8417_bbd95c04d017.slice/crio-bfe77bbe50cbb5b9748e9cb9ae7108bbf60c6f416373ca56e18f7d9c281fd8fb WatchSource:0}: Error finding container bfe77bbe50cbb5b9748e9cb9ae7108bbf60c6f416373ca56e18f7d9c281fd8fb: Status 404 returned error can't find the container with id bfe77bbe50cbb5b9748e9cb9ae7108bbf60c6f416373ca56e18f7d9c281fd8fb
Jul 29 21:26:05.682635 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:05.682550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" event={"ID":"3f0ed0ef-4ba9-488d-8417-bbd95c04d017","Type":"ContainerStarted","Data":"bfe77bbe50cbb5b9748e9cb9ae7108bbf60c6f416373ca56e18f7d9c281fd8fb"}
Jul 29 21:26:08.699039 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:08.699001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" event={"ID":"3f0ed0ef-4ba9-488d-8417-bbd95c04d017","Type":"ContainerStarted","Data":"0ef2cd70d4d90c24f4f716f6b679c7bd170f7f04c842cd59ab2033db987b669e"}
Jul 29 21:26:08.699458 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:08.699134    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:08.737635 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:08.737580    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" podStartSLOduration=1.075840396 podStartE2EDuration="3.737563984s" totalImagesPullingTime="2.661723588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:08.736507452 +0000 UTC m=+585.208086603" watchObservedRunningTime="2026-07-29 21:26:08.737563984 +0000 UTC m=+585.209143102"
Jul 29 21:26:19.705617 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:19.705586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:23.268945 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.268909    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:23.269397 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.269121    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" podUID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" containerName="limitador" containerID="cri-o://0ef2cd70d4d90c24f4f716f6b679c7bd170f7f04c842cd59ab2033db987b669e" gracePeriod=30
Jul 29 21:26:23.774740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.774708    2565 generic.go:356] "Generic (PLEG): container finished" podID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" containerID="0ef2cd70d4d90c24f4f716f6b679c7bd170f7f04c842cd59ab2033db987b669e" exitCode=0
Jul 29 21:26:23.774894 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.774775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" event={"ID":"3f0ed0ef-4ba9-488d-8417-bbd95c04d017","Type":"ContainerDied","Data":"0ef2cd70d4d90c24f4f716f6b679c7bd170f7f04c842cd59ab2033db987b669e"}
Jul 29 21:26:23.818334 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.818310    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:23.948919 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.948862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file\") pod \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") "
Jul 29 21:26:23.949157 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.948937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x295\" (UniqueName: \"kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295\") pod \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\" (UID: \"3f0ed0ef-4ba9-488d-8417-bbd95c04d017\") "
Jul 29 21:26:23.949241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.949220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file" (OuterVolumeSpecName: "config-file") pod "3f0ed0ef-4ba9-488d-8417-bbd95c04d017" (UID: "3f0ed0ef-4ba9-488d-8417-bbd95c04d017"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:26:23.949357 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.949334    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-config-file\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:26:23.951804 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:23.951742    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295" (OuterVolumeSpecName: "kube-api-access-8x295") pod "3f0ed0ef-4ba9-488d-8417-bbd95c04d017" (UID: "3f0ed0ef-4ba9-488d-8417-bbd95c04d017"). InnerVolumeSpecName "kube-api-access-8x295". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:26:24.049803 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.049763    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x295\" (UniqueName: \"kubernetes.io/projected/3f0ed0ef-4ba9-488d-8417-bbd95c04d017-kube-api-access-8x295\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:26:24.780024 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.779993    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"
Jul 29 21:26:24.780436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.779998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n7pxv" event={"ID":"3f0ed0ef-4ba9-488d-8417-bbd95c04d017","Type":"ContainerDied","Data":"bfe77bbe50cbb5b9748e9cb9ae7108bbf60c6f416373ca56e18f7d9c281fd8fb"}
Jul 29 21:26:24.780436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.780131    2565 scope.go:117] "RemoveContainer" containerID="0ef2cd70d4d90c24f4f716f6b679c7bd170f7f04c842cd59ab2033db987b669e"
Jul 29 21:26:24.797268 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.797229    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:24.798869 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:24.798849    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n7pxv"]
Jul 29 21:26:26.054059 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:26.054014    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" path="/var/lib/kubelet/pods/3f0ed0ef-4ba9-488d-8417-bbd95c04d017/volumes"
Jul 29 21:26:28.813307 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.813261    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"]
Jul 29 21:26:28.813780 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.813672    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" containerName="limitador"
Jul 29 21:26:28.813780 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.813685    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" containerName="limitador"
Jul 29 21:26:28.813780 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.813742    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f0ed0ef-4ba9-488d-8417-bbd95c04d017" containerName="limitador"
Jul 29 21:26:28.817991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.817969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:28.820424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.820395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 29 21:26:28.820424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.820421    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6c8z6\""
Jul 29 21:26:28.820610 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.820428    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 29 21:26:28.820610 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.820447    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 29 21:26:28.825608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.825586    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"]
Jul 29 21:26:28.994980 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.994944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6mc\" (UniqueName: \"kubernetes.io/projected/a597dacb-843c-4c6b-a375-9c0c89a43944-kube-api-access-7v6mc\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:28.995166 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:28.995012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a597dacb-843c-4c6b-a375-9c0c89a43944-data\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.095941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.095849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6mc\" (UniqueName: \"kubernetes.io/projected/a597dacb-843c-4c6b-a375-9c0c89a43944-kube-api-access-7v6mc\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.095941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.095940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a597dacb-843c-4c6b-a375-9c0c89a43944-data\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.096330 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.096312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a597dacb-843c-4c6b-a375-9c0c89a43944-data\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.104113 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.104091    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6mc\" (UniqueName: \"kubernetes.io/projected/a597dacb-843c-4c6b-a375-9c0c89a43944-kube-api-access-7v6mc\") pod \"postgres-69fb4594cf-rtk9h\" (UID: \"a597dacb-843c-4c6b-a375-9c0c89a43944\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.128622 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.128590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:29.257846 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.257813    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"]
Jul 29 21:26:29.260582 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:26:29.260556    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda597dacb_843c_4c6b_a375_9c0c89a43944.slice/crio-4bd7d6a7ab5974f28016443fa5b96bf5bd0907ef8ebf3a3a1edb21e328c2ca72 WatchSource:0}: Error finding container 4bd7d6a7ab5974f28016443fa5b96bf5bd0907ef8ebf3a3a1edb21e328c2ca72: Status 404 returned error can't find the container with id 4bd7d6a7ab5974f28016443fa5b96bf5bd0907ef8ebf3a3a1edb21e328c2ca72
Jul 29 21:26:29.806621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:29.806582    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h" event={"ID":"a597dacb-843c-4c6b-a375-9c0c89a43944","Type":"ContainerStarted","Data":"4bd7d6a7ab5974f28016443fa5b96bf5bd0907ef8ebf3a3a1edb21e328c2ca72"}
Jul 29 21:26:34.832990 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:34.832950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h" event={"ID":"a597dacb-843c-4c6b-a375-9c0c89a43944","Type":"ContainerStarted","Data":"54d3d1b153ab415aaa6cad74289450f94b376a166bd314e1e94be68a84da5341"}
Jul 29 21:26:34.833436 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:34.833074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:34.851832 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:34.851769    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h" podStartSLOduration=2.095894603 podStartE2EDuration="6.851746528s" totalImagesPullingTime="4.755851925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:34.848900557 +0000 UTC m=+611.320479651" watchObservedRunningTime="2026-07-29 21:26:34.851746528 +0000 UTC m=+611.323325604"
Jul 29 21:26:40.867732 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:40.867702    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rtk9h"
Jul 29 21:26:48.253606 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.253517    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:26:48.256886 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.256858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.259248 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.259218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:26:48.259521 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.259492    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:26:48.259638 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.259493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wh6vj\""
Jul 29 21:26:48.267849 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.267818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:26:48.370519 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.370472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjq5\" (UniqueName: \"kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.370742 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.370649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.427564 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.427530    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:26:48.437205 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.437176    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.441888 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.441860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:26:48.471738 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.471697    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.471909 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.471755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjq5\" (UniqueName: \"kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.471909 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:26:48.471854    2565 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 29 21:26:48.471984 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:26:48.471934    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert podName:19bba74f-ea04-45f8-a536-fdaffc3208e5 nodeName:}" failed. No retries permitted until 2026-07-29 21:26:48.971914971 +0000 UTC m=+625.443494026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert") pod "maas-controller-59995f6fc6-rtm2d" (UID: "19bba74f-ea04-45f8-a536-fdaffc3208e5") : secret "maas-controller-webhook-cert" not found
Jul 29 21:26:48.480531 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.480463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjq5\" (UniqueName: \"kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.572804 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.572706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.572961 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.572820    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njp2c\" (UniqueName: \"kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.674270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.674236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.674415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.674312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njp2c\" (UniqueName: \"kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.676674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.676647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.683421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.683397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njp2c\" (UniqueName: \"kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c\") pod \"maas-controller-656b7fbfd4-blr7c\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") " pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.749887 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.749824    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:48.886557 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.886532    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:26:48.888362 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:26:48.888336    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72151f11_728b_4452_b5f4_4d49b0289448.slice/crio-6e017ab82163742faaf628eaf023d57597c4ab9dae983a0c088dbfb136eab8fa WatchSource:0}: Error finding container 6e017ab82163742faaf628eaf023d57597c4ab9dae983a0c088dbfb136eab8fa: Status 404 returned error can't find the container with id 6e017ab82163742faaf628eaf023d57597c4ab9dae983a0c088dbfb136eab8fa
Jul 29 21:26:48.897648 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.897623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" event={"ID":"72151f11-728b-4452-b5f4-4d49b0289448","Type":"ContainerStarted","Data":"6e017ab82163742faaf628eaf023d57597c4ab9dae983a0c088dbfb136eab8fa"}
Jul 29 21:26:48.977520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.977489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:48.979819 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:48.979789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") pod \"maas-controller-59995f6fc6-rtm2d\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") " pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:49.168303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:49.168266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:49.306359 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:49.306328    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:26:49.308018 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:26:49.307991    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19bba74f_ea04_45f8_a536_fdaffc3208e5.slice/crio-04b7a7a6f40478912b5e3ae44c8acb2065eb3e55a02e4bd6fac94f9b9fcf1820 WatchSource:0}: Error finding container 04b7a7a6f40478912b5e3ae44c8acb2065eb3e55a02e4bd6fac94f9b9fcf1820: Status 404 returned error can't find the container with id 04b7a7a6f40478912b5e3ae44c8acb2065eb3e55a02e4bd6fac94f9b9fcf1820
Jul 29 21:26:49.905509 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:49.905468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" event={"ID":"19bba74f-ea04-45f8-a536-fdaffc3208e5","Type":"ContainerStarted","Data":"04b7a7a6f40478912b5e3ae44c8acb2065eb3e55a02e4bd6fac94f9b9fcf1820"}
Jul 29 21:26:51.917943 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:51.917908    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" event={"ID":"72151f11-728b-4452-b5f4-4d49b0289448","Type":"ContainerStarted","Data":"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"}
Jul 29 21:26:51.918469 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:51.917975    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:26:51.919226 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:51.919202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" event={"ID":"19bba74f-ea04-45f8-a536-fdaffc3208e5","Type":"ContainerStarted","Data":"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"}
Jul 29 21:26:51.919348 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:51.919327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:26:51.945926 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:26:51.945865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" podStartSLOduration=1.302269485 podStartE2EDuration="3.945846377s" totalImagesPullingTime="2.643576892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:51.94344911 +0000 UTC m=+628.415028204" watchObservedRunningTime="2026-07-29 21:26:51.945846377 +0000 UTC m=+628.417425454"
Jul 29 21:27:02.926783 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:02.926750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:27:02.927249 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:02.926966    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:27:02.944408 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:02.944349    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" podStartSLOduration=12.708095414 podStartE2EDuration="14.944332964s" totalImagesPullingTime="2.23623755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:26:51.965766934 +0000 UTC m=+628.437346012" watchObservedRunningTime="2026-07-29 21:27:02.944332964 +0000 UTC m=+639.415912039"
Jul 29 21:27:02.990305 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:02.990262    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:27:02.990520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:02.990493    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" podUID="19bba74f-ea04-45f8-a536-fdaffc3208e5" containerName="manager" containerID="cri-o://fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48" gracePeriod=30
Jul 29 21:27:03.247181 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.247144    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:27:03.411533 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.411486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvjq5\" (UniqueName: \"kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5\") pod \"19bba74f-ea04-45f8-a536-fdaffc3208e5\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") "
Jul 29 21:27:03.411736 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.411606    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") pod \"19bba74f-ea04-45f8-a536-fdaffc3208e5\" (UID: \"19bba74f-ea04-45f8-a536-fdaffc3208e5\") "
Jul 29 21:27:03.413807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.413768    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "19bba74f-ea04-45f8-a536-fdaffc3208e5" (UID: "19bba74f-ea04-45f8-a536-fdaffc3208e5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:27:03.413933 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.413825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5" (OuterVolumeSpecName: "kube-api-access-nvjq5") pod "19bba74f-ea04-45f8-a536-fdaffc3208e5" (UID: "19bba74f-ea04-45f8-a536-fdaffc3208e5"). InnerVolumeSpecName "kube-api-access-nvjq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:27:03.513363 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.513263    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/19bba74f-ea04-45f8-a536-fdaffc3208e5-webhook-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:27:03.513363 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.513302    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjq5\" (UniqueName: \"kubernetes.io/projected/19bba74f-ea04-45f8-a536-fdaffc3208e5-kube-api-access-nvjq5\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:27:03.975142 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.975105    2565 generic.go:356] "Generic (PLEG): container finished" podID="19bba74f-ea04-45f8-a536-fdaffc3208e5" containerID="fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48" exitCode=0
Jul 29 21:27:03.975535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.975146    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59995f6fc6-rtm2d"
Jul 29 21:27:03.975535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.975151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" event={"ID":"19bba74f-ea04-45f8-a536-fdaffc3208e5","Type":"ContainerDied","Data":"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"}
Jul 29 21:27:03.975535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.975271    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59995f6fc6-rtm2d" event={"ID":"19bba74f-ea04-45f8-a536-fdaffc3208e5","Type":"ContainerDied","Data":"04b7a7a6f40478912b5e3ae44c8acb2065eb3e55a02e4bd6fac94f9b9fcf1820"}
Jul 29 21:27:03.975535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.975302    2565 scope.go:117] "RemoveContainer" containerID="fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"
Jul 29 21:27:03.984943 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.984917    2565 scope.go:117] "RemoveContainer" containerID="fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"
Jul 29 21:27:03.985211 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:27:03.985194    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48\": container with ID starting with fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48 not found: ID does not exist" containerID="fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"
Jul 29 21:27:03.985284 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.985218    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48"} err="failed to get container status \"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48\": rpc error: code = NotFound desc = could not find container \"fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48\": container with ID starting with fe974e47bd0eeca341988a1d0d06d348ea63474f98f261b75bd4173e160cde48 not found: ID does not exist"
Jul 29 21:27:03.996968 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:03.996935    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:27:04.001725 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:04.001699    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59995f6fc6-rtm2d"]
Jul 29 21:27:04.055415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:04.055378    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19bba74f-ea04-45f8-a536-fdaffc3208e5" path="/var/lib/kubelet/pods/19bba74f-ea04-45f8-a536-fdaffc3208e5/volumes"
Jul 29 21:27:25.204171 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.204135    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"]
Jul 29 21:27:25.204582 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.204502    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19bba74f-ea04-45f8-a536-fdaffc3208e5" containerName="manager"
Jul 29 21:27:25.204582 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.204513    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="19bba74f-ea04-45f8-a536-fdaffc3208e5" containerName="manager"
Jul 29 21:27:25.204658 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.204583    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="19bba74f-ea04-45f8-a536-fdaffc3208e5" containerName="manager"
Jul 29 21:27:25.213347 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.213301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.215872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.215837    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 29 21:27:25.216015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.215919    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-p4qdf\""
Jul 29 21:27:25.219312 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.219286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"]
Jul 29 21:27:25.311619 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.311581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.311794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.311678    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6302d9-03b4-4d77-a431-7723f93e5ded-openshift-service-ca\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.311794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.311724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl9lv\" (UniqueName: \"kubernetes.io/projected/6d6302d9-03b4-4d77-a431-7723f93e5ded-kube-api-access-wl9lv\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.413037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.413002    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6302d9-03b4-4d77-a431-7723f93e5ded-openshift-service-ca\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.413236 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.413067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl9lv\" (UniqueName: \"kubernetes.io/projected/6d6302d9-03b4-4d77-a431-7723f93e5ded-kube-api-access-wl9lv\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.413236 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.413140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.413375 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:27:25.413277    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 29 21:27:25.413375 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:27:25.413369    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls podName:6d6302d9-03b4-4d77-a431-7723f93e5ded nodeName:}" failed. No retries permitted until 2026-07-29 21:27:25.913346237 +0000 UTC m=+662.384925310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls") pod "maas-api-79659c9889-xnwzp" (UID: "6d6302d9-03b4-4d77-a431-7723f93e5ded") : secret "maas-api-serving-cert" not found
Jul 29 21:27:25.413831 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.413804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d6302d9-03b4-4d77-a431-7723f93e5ded-openshift-service-ca\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.422024 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.421997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl9lv\" (UniqueName: \"kubernetes.io/projected/6d6302d9-03b4-4d77-a431-7723f93e5ded-kube-api-access-wl9lv\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.746567 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.746528    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2kvc7"]
Jul 29 21:27:25.750768 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.750742    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.754024 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.753997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-kdh6t\""
Jul 29 21:27:25.754145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.754009    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 29 21:27:25.762454 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.762424    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2kvc7"]
Jul 29 21:27:25.808593 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.808556    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"]
Jul 29 21:27:25.812555 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.812529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:25.818259 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.818196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8m6\" (UniqueName: \"kubernetes.io/projected/d0070d94-0d67-4f4d-9292-78b305332e4e-kube-api-access-hf8m6\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.818446 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.818357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0070d94-0d67-4f4d-9292-78b305332e4e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.830380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.830344    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"]
Jul 29 21:27:25.918960 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.918903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.918960 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.918960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8m6\" (UniqueName: \"kubernetes.io/projected/d0070d94-0d67-4f4d-9292-78b305332e4e-kube-api-access-hf8m6\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.919250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.919034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0070d94-0d67-4f4d-9292-78b305332e4e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.919393 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.919369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d7037b-7db0-4b22-ac71-33a4fc9d2973-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:25.919462 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.919433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68vn\" (UniqueName: \"kubernetes.io/projected/44d7037b-7db0-4b22-ac71-33a4fc9d2973-kube-api-access-b68vn\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:25.920181 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.920157    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0070d94-0d67-4f4d-9292-78b305332e4e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:25.922006 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.921983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d6302d9-03b4-4d77-a431-7723f93e5ded-maas-api-tls\") pod \"maas-api-79659c9889-xnwzp\" (UID: \"6d6302d9-03b4-4d77-a431-7723f93e5ded\") " pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:25.927605 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:25.927573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8m6\" (UniqueName: \"kubernetes.io/projected/d0070d94-0d67-4f4d-9292-78b305332e4e-kube-api-access-hf8m6\") pod \"payload-processing-5cc6cd444b-2kvc7\" (UID: \"d0070d94-0d67-4f4d-9292-78b305332e4e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:26.020962 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.020847    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d7037b-7db0-4b22-ac71-33a4fc9d2973-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:26.020962 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.020926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68vn\" (UniqueName: \"kubernetes.io/projected/44d7037b-7db0-4b22-ac71-33a4fc9d2973-kube-api-access-b68vn\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:26.021707 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.021682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d7037b-7db0-4b22-ac71-33a4fc9d2973-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:26.029994 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.029959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68vn\" (UniqueName: \"kubernetes.io/projected/44d7037b-7db0-4b22-ac71-33a4fc9d2973-kube-api-access-b68vn\") pod \"payload-pre-processing-687dbb6cf6-pt8x4\" (UID: \"44d7037b-7db0-4b22-ac71-33a4fc9d2973\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:26.062282 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.062243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:26.122538 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.122504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:26.126774 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.125973    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:26.229833 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.229793    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-2kvc7"]
Jul 29 21:27:26.232549 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:27:26.232516    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0070d94_0d67_4f4d_9292_78b305332e4e.slice/crio-38d17472e3e29f5f4c9f53948670defb82fe0ee3c98f172af057713da8bd2620 WatchSource:0}: Error finding container 38d17472e3e29f5f4c9f53948670defb82fe0ee3c98f172af057713da8bd2620: Status 404 returned error can't find the container with id 38d17472e3e29f5f4c9f53948670defb82fe0ee3c98f172af057713da8bd2620
Jul 29 21:27:26.235064 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.234884    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:27:26.314161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.313526    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"]
Jul 29 21:27:26.317460 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:27:26.317412    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6302d9_03b4_4d77_a431_7723f93e5ded.slice/crio-baf879f7306c97a3ed188e39672806e1e712f5e76eed4f16ba7edfe865d45c73 WatchSource:0}: Error finding container baf879f7306c97a3ed188e39672806e1e712f5e76eed4f16ba7edfe865d45c73: Status 404 returned error can't find the container with id baf879f7306c97a3ed188e39672806e1e712f5e76eed4f16ba7edfe865d45c73
Jul 29 21:27:26.367913 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:26.367818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"]
Jul 29 21:27:27.089668 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:27.089626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp" event={"ID":"6d6302d9-03b4-4d77-a431-7723f93e5ded","Type":"ContainerStarted","Data":"baf879f7306c97a3ed188e39672806e1e712f5e76eed4f16ba7edfe865d45c73"}
Jul 29 21:27:27.091432 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:27.091404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4" event={"ID":"44d7037b-7db0-4b22-ac71-33a4fc9d2973","Type":"ContainerStarted","Data":"627132803bdfdbd49d220bc4b749606adeeeb663cf1491920181bab64d3efa58"}
Jul 29 21:27:27.093408 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:27.093379    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7" event={"ID":"d0070d94-0d67-4f4d-9292-78b305332e4e","Type":"ContainerStarted","Data":"38d17472e3e29f5f4c9f53948670defb82fe0ee3c98f172af057713da8bd2620"}
Jul 29 21:27:30.118140 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.118103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp" event={"ID":"6d6302d9-03b4-4d77-a431-7723f93e5ded","Type":"ContainerStarted","Data":"72f2c92ca4dc2d5e79c55a92c0cc16bec3965eb9471eff93d229df46d57f84e8"}
Jul 29 21:27:30.118600 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.118189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:30.119699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.119677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4" event={"ID":"44d7037b-7db0-4b22-ac71-33a4fc9d2973","Type":"ContainerStarted","Data":"086237ebc1dfc49be4ba17e51f90f90b3b81d0383df1be4a80ff0829085062ce"}
Jul 29 21:27:30.119765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.119729    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:30.121232 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.121205    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7" event={"ID":"d0070d94-0d67-4f4d-9292-78b305332e4e","Type":"ContainerStarted","Data":"14bb1b35939adb54cd0cdaa11f9ad090328bb7616a45cc9ca6412a5b4d874e1c"}
Jul 29 21:27:30.121349 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.121251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:30.136320 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.136268    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp" podStartSLOduration=2.32325719 podStartE2EDuration="5.136251873s" totalImagesPullingTime="2.812994683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:30.134207706 +0000 UTC m=+666.605786784" watchObservedRunningTime="2026-07-29 21:27:30.136251873 +0000 UTC m=+666.607830976"
Jul 29 21:27:30.178108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.172980    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4" podStartSLOduration=2.406409343 podStartE2EDuration="5.172954323s" totalImagesPullingTime="2.76654498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:30.166451284 +0000 UTC m=+666.638030415" watchObservedRunningTime="2026-07-29 21:27:30.172954323 +0000 UTC m=+666.644533402"
Jul 29 21:27:30.195620 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:30.195513    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7" podStartSLOduration=2.2981888440000002 podStartE2EDuration="5.195497673s" totalImagesPullingTime="2.897308829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:30.195284872 +0000 UTC m=+666.666863938" watchObservedRunningTime="2026-07-29 21:27:30.195497673 +0000 UTC m=+666.667076748"
Jul 29 21:27:34.142438 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.142403    2565 generic.go:356] "Generic (PLEG): container finished" podID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerID="cd640e11cd982c20a8e27c7968f5b6250809deef67ca91e42552b8778b8844be" exitCode=0
Jul 29 21:27:34.142851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.142472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" event={"ID":"3bc54791-82ee-46cc-bc56-3497049bfef7","Type":"ContainerDied","Data":"cd640e11cd982c20a8e27c7968f5b6250809deef67ca91e42552b8778b8844be"}
Jul 29 21:27:34.142895 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.142868    2565 scope.go:117] "RemoveContainer" containerID="cd640e11cd982c20a8e27c7968f5b6250809deef67ca91e42552b8778b8844be"
Jul 29 21:27:34.143820 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.143789    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:27:34.143923 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.143855    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:27:34.143923 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:34.143886    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:27:35.152968 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:35.152923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" event={"ID":"3bc54791-82ee-46cc-bc56-3497049bfef7","Type":"ContainerStarted","Data":"c55d00d32f258c5fe5021f05aa68b48610494325aee72ee1971d6e2c0214ed65"}
Jul 29 21:27:35.368321 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:35.368283    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:27:35.369666 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:35.369637    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mjv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 29 21:27:35.369791 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:35.369698    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podUID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 29 21:27:36.131795 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:36.131760    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-79659c9889-xnwzp"
Jul 29 21:27:36.367898 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:36.367861    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mjv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 29 21:27:36.368293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:36.367929    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podUID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 29 21:27:37.368821 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:37.368786    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mjv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 29 21:27:37.369244 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:37.368857    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podUID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 29 21:27:37.625744 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:37.625652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:27:38.367984 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:38.367949    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-mjv7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body=
Jul 29 21:27:38.368168 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:38.368018    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r" podUID="3bc54791-82ee-46cc-bc56-3497049bfef7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused"
Jul 29 21:27:39.372571 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:39.372538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:27:39.373652 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:39.373631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-mjv7r"
Jul 29 21:27:41.128673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:41.128639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-2kvc7"
Jul 29 21:27:41.129081 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:41.128722    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-pt8x4"
Jul 29 21:27:47.736245 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.736203    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"]
Jul 29 21:27:47.741549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.741518    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.744501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.744449    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 29 21:27:47.745345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.745306    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 29 21:27:47.745345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.745311    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 29 21:27:47.745539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.745364    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8bjkc\""
Jul 29 21:27:47.753723 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.753693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"]
Jul 29 21:27:47.854688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.854871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.854871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.854871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854851    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.855008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8n9\" (UniqueName: \"kubernetes.io/projected/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-kube-api-access-vz8n9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.855008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.854933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.955853 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.955816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.955873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8n9\" (UniqueName: \"kubernetes.io/projected/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-kube-api-access-vz8n9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.955905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.955968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.955993    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956093 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.956014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956365 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.956343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.956355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.956452 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.956426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.958272 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.958247    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.958430 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.958412    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:47.964619 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:47.964590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8n9\" (UniqueName: \"kubernetes.io/projected/ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a-kube-api-access-vz8n9\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8\" (UID: \"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:48.053742 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:48.053651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:48.204896 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:48.204860    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"]
Jul 29 21:27:48.207864 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:27:48.207835    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4a5a4d_45d7_45c3_8ec8_f211a0a8b27a.slice/crio-2b13e38d50f13142f9818b4fa1dac1f052f7ab762fa889637ed98997343e5b5f WatchSource:0}: Error finding container 2b13e38d50f13142f9818b4fa1dac1f052f7ab762fa889637ed98997343e5b5f: Status 404 returned error can't find the container with id 2b13e38d50f13142f9818b4fa1dac1f052f7ab762fa889637ed98997343e5b5f
Jul 29 21:27:48.233554 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:48.233513    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8" event={"ID":"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a","Type":"ContainerStarted","Data":"2b13e38d50f13142f9818b4fa1dac1f052f7ab762fa889637ed98997343e5b5f"}
Jul 29 21:27:50.246935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:50.246887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8" event={"ID":"ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a","Type":"ContainerStarted","Data":"bf7ec16a53052541a8e4dfef8c41dfcaa0d8c5400f9c5d17c1bf2b9f21a8db68"}
Jul 29 21:27:50.269292 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:50.269231    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8" podStartSLOduration=2.128892565 podStartE2EDuration="3.269212804s" totalImagesPullingTime="1.140320239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:27:50.267234117 +0000 UTC m=+686.738813208" watchObservedRunningTime="2026-07-29 21:27:50.269212804 +0000 UTC m=+686.740791880"
Jul 29 21:27:58.055120 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:58.055088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:58.055592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:58.055139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:58.066755 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:58.066726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:27:58.298372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:27:58.298329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8"
Jul 29 21:28:12.213704 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.213666    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"]
Jul 29 21:28:12.219931 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.219910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.222600 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.222578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 29 21:28:12.230420 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.230391    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"]
Jul 29 21:28:12.288592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.288784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288705    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.288784 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288761    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.288917 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwx2\" (UniqueName: \"kubernetes.io/projected/7c1a61af-d182-44ad-afd4-ac75e1e04e50-kube-api-access-bfwx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.288917 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.289027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.288959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwx2\" (UniqueName: \"kubernetes.io/projected/7c1a61af-d182-44ad-afd4-ac75e1e04e50-kube-api-access-bfwx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390575 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390552    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390899 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390981 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390885    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.390981 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.390951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.392631 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.392610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1a61af-d182-44ad-afd4-ac75e1e04e50-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.393163 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.393143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1a61af-d182-44ad-afd4-ac75e1e04e50-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.400536 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.400506    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwx2\" (UniqueName: \"kubernetes.io/projected/7c1a61af-d182-44ad-afd4-ac75e1e04e50-kube-api-access-bfwx2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-j4tnd\" (UID: \"7c1a61af-d182-44ad-afd4-ac75e1e04e50\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.530591 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.530498    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:12.670540 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:12.670512    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"]
Jul 29 21:28:12.672143 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:28:12.672112    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1a61af_d182_44ad_afd4_ac75e1e04e50.slice/crio-49e2a30e3fb1b3779d445180182dbdf3b370b0dab2c3024764dae192988831b8 WatchSource:0}: Error finding container 49e2a30e3fb1b3779d445180182dbdf3b370b0dab2c3024764dae192988831b8: Status 404 returned error can't find the container with id 49e2a30e3fb1b3779d445180182dbdf3b370b0dab2c3024764dae192988831b8
Jul 29 21:28:13.357376 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:13.357329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd" event={"ID":"7c1a61af-d182-44ad-afd4-ac75e1e04e50","Type":"ContainerStarted","Data":"e3adb00da0335960eb13c137a984330cea9fea94fe8545fb0c370fdb04a72883"}
Jul 29 21:28:13.357765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:13.357427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd" event={"ID":"7c1a61af-d182-44ad-afd4-ac75e1e04e50","Type":"ContainerStarted","Data":"49e2a30e3fb1b3779d445180182dbdf3b370b0dab2c3024764dae192988831b8"}
Jul 29 21:28:13.384579 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:13.384522    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd" podStartSLOduration=1.220755177 podStartE2EDuration="1.384506535s" totalImagesPullingTime="163.751358ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:28:13.38039783 +0000 UTC m=+709.851976905" watchObservedRunningTime="2026-07-29 21:28:13.384506535 +0000 UTC m=+709.856085610"
Jul 29 21:28:22.531327 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:22.531244    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:22.531327 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:22.531313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:22.543729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:22.543703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:23.416765 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:23.416733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-j4tnd"
Jul 29 21:28:47.409485 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.409443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"]
Jul 29 21:28:47.414406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.414383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.417230 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.417208    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 29 21:28:47.425720 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.425686    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"]
Jul 29 21:28:47.517802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.517764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.517802 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.517806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.517997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.517823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b962\" (UniqueName: \"kubernetes.io/projected/0735801d-f5dc-411f-b5db-7aad83eb8922-kube-api-access-8b962\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.517997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.517954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.518094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.518008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.518130 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.518112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0735801d-f5dc-411f-b5db-7aad83eb8922-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619174 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0735801d-f5dc-411f-b5db-7aad83eb8922-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619372 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b962\" (UniqueName: \"kubernetes.io/projected/0735801d-f5dc-411f-b5db-7aad83eb8922-kube-api-access-8b962\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619695 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619672    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619782 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619757    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.619836 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.619786    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.621640 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.621618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0735801d-f5dc-411f-b5db-7aad83eb8922-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.622013 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.621991    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0735801d-f5dc-411f-b5db-7aad83eb8922-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.628212 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.628183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b962\" (UniqueName: \"kubernetes.io/projected/0735801d-f5dc-411f-b5db-7aad83eb8922-kube-api-access-8b962\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6dph8\" (UID: \"0735801d-f5dc-411f-b5db-7aad83eb8922\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.728008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.727969    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:47.870686 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:47.870652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"]
Jul 29 21:28:47.874958 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:28:47.874927    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0735801d_f5dc_411f_b5db_7aad83eb8922.slice/crio-60fc8bc729c9394f61f40d1068639f782df1cb17ff3facef0005e685ecb935f8 WatchSource:0}: Error finding container 60fc8bc729c9394f61f40d1068639f782df1cb17ff3facef0005e685ecb935f8: Status 404 returned error can't find the container with id 60fc8bc729c9394f61f40d1068639f782df1cb17ff3facef0005e685ecb935f8
Jul 29 21:28:48.530280 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:48.530244    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8" event={"ID":"0735801d-f5dc-411f-b5db-7aad83eb8922","Type":"ContainerStarted","Data":"21bea669e709c16cc34bb949164d0e9be48f849b552cc9384e5384af300ca590"}
Jul 29 21:28:48.530650 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:48.530296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8" event={"ID":"0735801d-f5dc-411f-b5db-7aad83eb8922","Type":"ContainerStarted","Data":"60fc8bc729c9394f61f40d1068639f782df1cb17ff3facef0005e685ecb935f8"}
Jul 29 21:28:48.549102 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:48.549032    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8" podStartSLOduration=1.356138306 podStartE2EDuration="1.549013225s" totalImagesPullingTime="192.874919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:28:48.547691145 +0000 UTC m=+745.019270244" watchObservedRunningTime="2026-07-29 21:28:48.549013225 +0000 UTC m=+745.020592300"
Jul 29 21:28:57.728920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:57.728871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:57.729451 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:57.728985    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:57.742273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:57.742245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:28:58.591642 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:28:58.591614    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6dph8"
Jul 29 21:30:00.167891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.167809    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"]
Jul 29 21:30:00.171407 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.171389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"
Jul 29 21:30:00.178684 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.178660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\""
Jul 29 21:30:00.181070 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.181033    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"]
Jul 29 21:30:00.181759 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.181739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"
Jul 29 21:30:00.320083 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.320019    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"]
Jul 29 21:30:00.322656 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:30:00.322632    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f9bfda_4143_4878_85d2_b455b5402371.slice/crio-596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30 WatchSource:0}: Error finding container 596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30: Status 404 returned error can't find the container with id 596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30
Jul 29 21:30:00.886014 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:00.885979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286" event={"ID":"25f9bfda-4143-4878-85d2-b455b5402371","Type":"ContainerStarted","Data":"596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30"}
Jul 29 21:30:03.902316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:03.902230    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286" event={"ID":"25f9bfda-4143-4878-85d2-b455b5402371","Type":"ContainerStarted","Data":"c9d4f248dbddd7fbd9c0edf094b8aaf8a748dcdca450e784bc130eaa4a8d862a"}
Jul 29 21:30:03.919255 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:03.919201    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286" podStartSLOduration=0.878862417 podStartE2EDuration="3.919183656s" totalImagesPullingTime="3.040321239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:30:03.916869533 +0000 UTC m=+820.388448642" watchObservedRunningTime="2026-07-29 21:30:03.919183656 +0000 UTC m=+820.390762732"
Jul 29 21:30:04.908960 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:04.908921    2565 generic.go:356] "Generic (PLEG): container finished" podID="25f9bfda-4143-4878-85d2-b455b5402371" containerID="c9d4f248dbddd7fbd9c0edf094b8aaf8a748dcdca450e784bc130eaa4a8d862a" exitCode=0
Jul 29 21:30:04.909384 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:04.909001    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286" event={"ID":"25f9bfda-4143-4878-85d2-b455b5402371","Type":"ContainerDied","Data":"c9d4f248dbddd7fbd9c0edf094b8aaf8a748dcdca450e784bc130eaa4a8d862a"}
Jul 29 21:30:06.050462 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:06.050428    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"
Jul 29 21:30:06.919844 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:06.919806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286" event={"ID":"25f9bfda-4143-4878-85d2-b455b5402371","Type":"ContainerDied","Data":"596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30"}
Jul 29 21:30:06.920017 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:06.919858    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="596f5b8af9cef3e86d15b49a251f97df954bd1976f5ef0bf6e7d801cad92ca30"
Jul 29 21:30:06.920017 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:06.919815    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"
Jul 29 21:30:45.144926 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.144889    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:30:45.145520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.145113    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" podUID="72151f11-728b-4452-b5f4-4d49b0289448" containerName="manager" containerID="cri-o://db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a" gracePeriod=30
Jul 29 21:30:45.397463 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.397397    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:30:45.493316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.493259    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert\") pod \"72151f11-728b-4452-b5f4-4d49b0289448\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") "
Jul 29 21:30:45.493506 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.493356    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njp2c\" (UniqueName: \"kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c\") pod \"72151f11-728b-4452-b5f4-4d49b0289448\" (UID: \"72151f11-728b-4452-b5f4-4d49b0289448\") "
Jul 29 21:30:45.495317 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.495283    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c" (OuterVolumeSpecName: "kube-api-access-njp2c") pod "72151f11-728b-4452-b5f4-4d49b0289448" (UID: "72151f11-728b-4452-b5f4-4d49b0289448"). InnerVolumeSpecName "kube-api-access-njp2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:30:45.495428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.495401    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "72151f11-728b-4452-b5f4-4d49b0289448" (UID: "72151f11-728b-4452-b5f4-4d49b0289448"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:30:45.594073 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.594012    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njp2c\" (UniqueName: \"kubernetes.io/projected/72151f11-728b-4452-b5f4-4d49b0289448-kube-api-access-njp2c\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:45.594073 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:45.594070    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/72151f11-728b-4452-b5f4-4d49b0289448-webhook-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:30:46.106954 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.106922    2565 generic.go:356] "Generic (PLEG): container finished" podID="72151f11-728b-4452-b5f4-4d49b0289448" containerID="db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a" exitCode=0
Jul 29 21:30:46.107144 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.106969    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-blr7c"
Jul 29 21:30:46.107144 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.106979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" event={"ID":"72151f11-728b-4452-b5f4-4d49b0289448","Type":"ContainerDied","Data":"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"}
Jul 29 21:30:46.107144 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.107019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-blr7c" event={"ID":"72151f11-728b-4452-b5f4-4d49b0289448","Type":"ContainerDied","Data":"6e017ab82163742faaf628eaf023d57597c4ab9dae983a0c088dbfb136eab8fa"}
Jul 29 21:30:46.107144 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.107063    2565 scope.go:117] "RemoveContainer" containerID="db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"
Jul 29 21:30:46.116094 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.116071    2565 scope.go:117] "RemoveContainer" containerID="db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"
Jul 29 21:30:46.116374 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:30:46.116355    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a\": container with ID starting with db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a not found: ID does not exist" containerID="db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"
Jul 29 21:30:46.116424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.116383    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a"} err="failed to get container status \"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a\": rpc error: code = NotFound desc = could not find container \"db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a\": container with ID starting with db8b4c21c53ce6e4e90ead270408661b68f6dc6c4acb20308fd81bd8feb2f08a not found: ID does not exist"
Jul 29 21:30:46.126480 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.126454    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:30:46.130985 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:46.130960    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-blr7c"]
Jul 29 21:30:48.054943 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:30:48.054884    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72151f11-728b-4452-b5f4-4d49b0289448" path="/var/lib/kubelet/pods/72151f11-728b-4452-b5f4-4d49b0289448/volumes"
Jul 29 21:31:46.921026 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.920987    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:31:46.928429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928386    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25f9bfda-4143-4878-85d2-b455b5402371" containerName="cleanup"
Jul 29 21:31:46.928429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928428    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25f9bfda-4143-4878-85d2-b455b5402371" containerName="cleanup"
Jul 29 21:31:46.928730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928493    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72151f11-728b-4452-b5f4-4d49b0289448" containerName="manager"
Jul 29 21:31:46.928730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928503    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="72151f11-728b-4452-b5f4-4d49b0289448" containerName="manager"
Jul 29 21:31:46.928924 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928902    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="72151f11-728b-4452-b5f4-4d49b0289448" containerName="manager"
Jul 29 21:31:46.928997 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.928926    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25f9bfda-4143-4878-85d2-b455b5402371" containerName="cleanup"
Jul 29 21:31:46.932970 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.932944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:31:46.933140 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.933088    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:46.937136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.937078    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:31:46.937284 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.937189    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:31:46.937798 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.937087    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wh6vj\""
Jul 29 21:31:46.988963 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.988929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9qfd\" (UniqueName: \"kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:46.989223 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:46.989021    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.089551 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.089506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.089748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.089597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9qfd\" (UniqueName: \"kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.092334 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.092301    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.099542 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.099504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9qfd\" (UniqueName: \"kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd\") pod \"maas-controller-656b7fbfd4-s4h7d\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") " pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.246310 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.246214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:47.392729 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.392699    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:31:47.394606 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:31:47.394576    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc9fd4f_c569_4c90_8f7b_271619cdf7dd.slice/crio-628fccafd74d5c35e8a81cd1de17330a38e195e5de32072c07b2f375acb128df WatchSource:0}: Error finding container 628fccafd74d5c35e8a81cd1de17330a38e195e5de32072c07b2f375acb128df: Status 404 returned error can't find the container with id 628fccafd74d5c35e8a81cd1de17330a38e195e5de32072c07b2f375acb128df
Jul 29 21:31:47.406886 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:47.406850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" event={"ID":"adc9fd4f-c569-4c90-8f7b-271619cdf7dd","Type":"ContainerStarted","Data":"628fccafd74d5c35e8a81cd1de17330a38e195e5de32072c07b2f375acb128df"}
Jul 29 21:31:48.414359 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:48.414261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" event={"ID":"adc9fd4f-c569-4c90-8f7b-271619cdf7dd","Type":"ContainerStarted","Data":"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"}
Jul 29 21:31:48.414800 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:48.414392    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:31:48.435640 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:48.435586    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" podStartSLOduration=1.816455897 podStartE2EDuration="2.435569276s" totalImagesPullingTime="619.113379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:31:48.432706973 +0000 UTC m=+924.904286050" watchObservedRunningTime="2026-07-29 21:31:48.435569276 +0000 UTC m=+924.907148351"
Jul 29 21:31:59.421322 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:31:59.421289    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:32:04.066471 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.066434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:32:04.066852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.066611    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" podUID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" containerName="manager" containerID="cri-o://a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b" gracePeriod=30
Jul 29 21:32:04.323311 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.323243    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:32:04.456203 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.456163    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert\") pod \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") "
Jul 29 21:32:04.456401 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.456293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9qfd\" (UniqueName: \"kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd\") pod \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\" (UID: \"adc9fd4f-c569-4c90-8f7b-271619cdf7dd\") "
Jul 29 21:32:04.463993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.458975    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "adc9fd4f-c569-4c90-8f7b-271619cdf7dd" (UID: "adc9fd4f-c569-4c90-8f7b-271619cdf7dd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:32:04.464507 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.464478    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd" (OuterVolumeSpecName: "kube-api-access-j9qfd") pod "adc9fd4f-c569-4c90-8f7b-271619cdf7dd" (UID: "adc9fd4f-c569-4c90-8f7b-271619cdf7dd"). InnerVolumeSpecName "kube-api-access-j9qfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:32:04.493978 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.493944    2565 generic.go:356] "Generic (PLEG): container finished" podID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" containerID="a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b" exitCode=0
Jul 29 21:32:04.494159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.494002    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d"
Jul 29 21:32:04.494159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.494024    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" event={"ID":"adc9fd4f-c569-4c90-8f7b-271619cdf7dd","Type":"ContainerDied","Data":"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"}
Jul 29 21:32:04.494159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.494122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-s4h7d" event={"ID":"adc9fd4f-c569-4c90-8f7b-271619cdf7dd","Type":"ContainerDied","Data":"628fccafd74d5c35e8a81cd1de17330a38e195e5de32072c07b2f375acb128df"}
Jul 29 21:32:04.494159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.494146    2565 scope.go:117] "RemoveContainer" containerID="a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"
Jul 29 21:32:04.503306 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.503289    2565 scope.go:117] "RemoveContainer" containerID="a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"
Jul 29 21:32:04.503572 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:32:04.503552    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b\": container with ID starting with a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b not found: ID does not exist" containerID="a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"
Jul 29 21:32:04.503611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.503581    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b"} err="failed to get container status \"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b\": rpc error: code = NotFound desc = could not find container \"a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b\": container with ID starting with a8466ddc022ae6003feec568a3965c57b8f1d95d4ebc52cd4d76a9bbe7dbd55b not found: ID does not exist"
Jul 29 21:32:04.516502 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.516475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:32:04.521150 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.521129    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-s4h7d"]
Jul 29 21:32:04.558156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.558121    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9qfd\" (UniqueName: \"kubernetes.io/projected/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-kube-api-access-j9qfd\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:32:04.558156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:04.558152    2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc9fd4f-c569-4c90-8f7b-271619cdf7dd-webhook-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:32:05.310847 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.310815    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-v4htm"]
Jul 29 21:32:05.311253 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.311237    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" containerName="manager"
Jul 29 21:32:05.311292 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.311255    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" containerName="manager"
Jul 29 21:32:05.311350 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.311337    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" containerName="manager"
Jul 29 21:32:05.315528 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.315509    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.318111 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.318086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wh6vj\""
Jul 29 21:32:05.318239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.318131    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 29 21:32:05.318949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.318929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 29 21:32:05.329745 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.329720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-v4htm"]
Jul 29 21:32:05.467812 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.467774    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b7602e9-e020-45e5-b61c-e733d74d6067-webhook-cert\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.468010 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.467896    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlgc\" (UniqueName: \"kubernetes.io/projected/9b7602e9-e020-45e5-b61c-e733d74d6067-kube-api-access-ktlgc\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.568680 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.568591    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b7602e9-e020-45e5-b61c-e733d74d6067-webhook-cert\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.568831 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.568710    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlgc\" (UniqueName: \"kubernetes.io/projected/9b7602e9-e020-45e5-b61c-e733d74d6067-kube-api-access-ktlgc\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.571226 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.571206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9b7602e9-e020-45e5-b61c-e733d74d6067-webhook-cert\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.577275 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.577255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlgc\" (UniqueName: \"kubernetes.io/projected/9b7602e9-e020-45e5-b61c-e733d74d6067-kube-api-access-ktlgc\") pod \"maas-controller-656b7fbfd4-v4htm\" (UID: \"9b7602e9-e020-45e5-b61c-e733d74d6067\") " pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.626085 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.626031    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:05.770852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:05.770827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b7fbfd4-v4htm"]
Jul 29 21:32:05.772872 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:32:05.772841    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7602e9_e020_45e5_b61c_e733d74d6067.slice/crio-e217709f109072712f5537e7e817e950abd6a37f8c27921e3d6a09f0af1f852f WatchSource:0}: Error finding container e217709f109072712f5537e7e817e950abd6a37f8c27921e3d6a09f0af1f852f: Status 404 returned error can't find the container with id e217709f109072712f5537e7e817e950abd6a37f8c27921e3d6a09f0af1f852f
Jul 29 21:32:06.055367 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:06.055332    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc9fd4f-c569-4c90-8f7b-271619cdf7dd" path="/var/lib/kubelet/pods/adc9fd4f-c569-4c90-8f7b-271619cdf7dd/volumes"
Jul 29 21:32:06.506554 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:06.506520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-v4htm" event={"ID":"9b7602e9-e020-45e5-b61c-e733d74d6067","Type":"ContainerStarted","Data":"289261ace72be520c4c79fd7d854584dce13eef54c957267d8c3f5e761bd2c6d"}
Jul 29 21:32:06.506901 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:06.506575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b7fbfd4-v4htm" event={"ID":"9b7602e9-e020-45e5-b61c-e733d74d6067","Type":"ContainerStarted","Data":"e217709f109072712f5537e7e817e950abd6a37f8c27921e3d6a09f0af1f852f"}
Jul 29 21:32:06.506901 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:06.506648    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:32:06.524738 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:06.524683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b7fbfd4-v4htm" podStartSLOduration=0.93977927 podStartE2EDuration="1.524667276s" totalImagesPullingTime="584.888006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:32:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:32:06.523036608 +0000 UTC m=+942.994615685" watchObservedRunningTime="2026-07-29 21:32:06.524667276 +0000 UTC m=+942.996246379"
Jul 29 21:32:17.513950 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:32:17.513916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b7fbfd4-v4htm"
Jul 29 21:45:00.139631 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.139594    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"]
Jul 29 21:45:00.143342 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.143323    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"
Jul 29 21:45:00.147820 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.147799    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\""
Jul 29 21:45:00.152399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.152373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"]
Jul 29 21:45:00.155262 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.155240    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"
Jul 29 21:45:00.301232 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.301199    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"]
Jul 29 21:45:00.304196 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:45:00.304151    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc4f2e7_dfa1_452d_93b0_4c3b39cba2c7.slice/crio-22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe WatchSource:0}: Error finding container 22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe: Status 404 returned error can't find the container with id 22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe
Jul 29 21:45:00.332899 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:00.332872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh" event={"ID":"8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7","Type":"ContainerStarted","Data":"22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe"}
Jul 29 21:45:01.339828 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:01.339726    2565 generic.go:356] "Generic (PLEG): container finished" podID="8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7" containerID="45f11266245cdacee84d7755754c69bd63c627a287b5d00c37c2643ca3db3604" exitCode=0
Jul 29 21:45:01.339828 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:01.339785    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh" event={"ID":"8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7","Type":"ContainerDied","Data":"45f11266245cdacee84d7755754c69bd63c627a287b5d00c37c2643ca3db3604"}
Jul 29 21:45:02.479723 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:02.479699    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"
Jul 29 21:45:03.351339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:03.351294    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh"
Jul 29 21:45:03.351339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:03.351329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756025-nqhwh" event={"ID":"8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7","Type":"ContainerDied","Data":"22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe"}
Jul 29 21:45:03.351560 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:45:03.351382    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22ac9d3c659f63a969e533125711327d36f22bdfc45980151d74a92b1cf679fe"
Jul 29 21:49:19.440096 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.439991    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:49:19.440552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.440484    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7" containerName="cleanup"
Jul 29 21:49:19.440552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.440499    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7" containerName="cleanup"
Jul 29 21:49:19.440628 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.440574    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7" containerName="cleanup"
Jul 29 21:49:19.444139 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.444115    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.450011 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.449987    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-972150f2-openshift-default-dockercfg-c7ggq\""
Jul 29 21:49:19.498807 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.498772    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:49:19.550417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2b8\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550607    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550763 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550763 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.550763 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.550675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652178 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652352 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652352 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652352 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2b8\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652803 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652635    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652803 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.652918 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652894    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.653011 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.652993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.653108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.653006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.658074 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.655009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.658074 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.655431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.668626 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.668595    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.668880 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.668860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2b8\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8\") pod \"e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") " pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.754552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.754454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:19.913940 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.913914    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:49:19.914917 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:49:19.914891    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140e640e_9a7d_4995_adab_5c2b64bf3808.slice/crio-446928fa473d91a8e6b9b78a5364bc1eace5f19aef045efb1b59f7c73103deac WatchSource:0}: Error finding container 446928fa473d91a8e6b9b78a5364bc1eace5f19aef045efb1b59f7c73103deac: Status 404 returned error can't find the container with id 446928fa473d91a8e6b9b78a5364bc1eace5f19aef045efb1b59f7c73103deac
Jul 29 21:49:19.917078 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.917030    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:19.917147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.917110    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:19.917147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:19.917140    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:20.648658 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:20.648614    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerStarted","Data":"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"}
Jul 29 21:49:20.649075 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:20.648678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerStarted","Data":"446928fa473d91a8e6b9b78a5364bc1eace5f19aef045efb1b59f7c73103deac"}
Jul 29 21:49:20.689294 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:20.689236    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podStartSLOduration=1.689221278 podStartE2EDuration="1.689221278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:49:20.687483902 +0000 UTC m=+1977.159062990" watchObservedRunningTime="2026-07-29 21:49:20.689221278 +0000 UTC m=+1977.160800428"
Jul 29 21:49:20.755562 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:20.755528    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:20.761085 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:20.761039    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:21.654426 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:21.654390    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:21.655491 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:21.655467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:23.217902 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.217871    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:49:23.221549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.221532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.224266 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.224250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-972150f2-serving-cert\""
Jul 29 21:49:23.234943 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.234918    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:49:23.289278 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.289243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.289450 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.289280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q95xq\" (UniqueName: \"kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.289450 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.289324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.390512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.390477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.390512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.390513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q95xq\" (UniqueName: \"kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.390767 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.390542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.391323 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.391297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.392928 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.392903    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.408941 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.408904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q95xq\" (UniqueName: \"kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq\") pod \"maas-api-e2e-ait-972150f2-ffd8fd578-vlj29\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.532898 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.532794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:23.898031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.898002    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:49:23.900763 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:49:23.900717    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232631ed_9f5b_4686_aae0_5ba0e6c202ee.slice/crio-7ffe1426eaaad4229f8d2f7f25b2f4017afa2186b803d63712a1381e023c69e8 WatchSource:0}: Error finding container 7ffe1426eaaad4229f8d2f7f25b2f4017afa2186b803d63712a1381e023c69e8: Status 404 returned error can't find the container with id 7ffe1426eaaad4229f8d2f7f25b2f4017afa2186b803d63712a1381e023c69e8
Jul 29 21:49:23.902410 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:23.902388    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:49:24.672748 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:24.672709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" event={"ID":"232631ed-9f5b-4686-aae0-5ba0e6c202ee","Type":"ContainerStarted","Data":"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"}
Jul 29 21:49:24.673122 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:24.672772    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" event={"ID":"232631ed-9f5b-4686-aae0-5ba0e6c202ee","Type":"ContainerStarted","Data":"7ffe1426eaaad4229f8d2f7f25b2f4017afa2186b803d63712a1381e023c69e8"}
Jul 29 21:49:24.673122 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:24.672809    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:24.697092 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:24.697030    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" podStartSLOduration=1.192511154 podStartE2EDuration="1.696997558s" totalImagesPullingTime="504.486404ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:49:24.695330309 +0000 UTC m=+1981.166909397" watchObservedRunningTime="2026-07-29 21:49:24.696997558 +0000 UTC m=+1981.168576633"
Jul 29 21:49:30.683211 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:30.683180    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:49:37.696243 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:37.696197    2565 patch_prober.go:28] interesting pod/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 21:49:37.696637 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:37.696267    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 21:49:38.747990 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.747957    2565 generic.go:356] "Generic (PLEG): container finished" podID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerID="6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9" exitCode=0
Jul 29 21:49:38.748441 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.748051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerDied","Data":"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"}
Jul 29 21:49:38.748501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.748474    2565 scope.go:117] "RemoveContainer" containerID="6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"
Jul 29 21:49:38.750206 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.750150    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:38.755866 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.750327    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:38.755866 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:38.750516    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:49:39.758814 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:39.758774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerStarted","Data":"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"}
Jul 29 21:49:40.754685 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:40.754646    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:40.755175 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:40.755141    2565 patch_prober.go:28] interesting pod/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 29 21:49:40.755420 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:40.755390    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 29 21:49:41.755493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:41.755007    2565 patch_prober.go:28] interesting pod/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 29 21:49:41.755493 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:41.755107    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 29 21:49:42.755628 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:42.755577    2565 patch_prober.go:28] interesting pod/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 29 21:49:42.756147 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:42.755653    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 29 21:49:43.759097 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:43.759066    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:43.759550 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:43.759531    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:49:43.760254 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:49:43.760237    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:50:03.942403 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:03.942353    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:50:03.942940 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:03.942618    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" podUID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" containerName="maas-api" containerID="cri-o://0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202" gracePeriod=30
Jul 29 21:50:04.214929 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.214897    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:50:04.307850 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.307811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q95xq\" (UniqueName: \"kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq\") pod \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") "
Jul 29 21:50:04.308022 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.307859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls\") pod \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") "
Jul 29 21:50:04.308022 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.307898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca\") pod \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\" (UID: \"232631ed-9f5b-4686-aae0-5ba0e6c202ee\") "
Jul 29 21:50:04.308346 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.308309    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "232631ed-9f5b-4686-aae0-5ba0e6c202ee" (UID: "232631ed-9f5b-4686-aae0-5ba0e6c202ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:04.310218 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.310192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "232631ed-9f5b-4686-aae0-5ba0e6c202ee" (UID: "232631ed-9f5b-4686-aae0-5ba0e6c202ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:50:04.310326 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.310237    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq" (OuterVolumeSpecName: "kube-api-access-q95xq") pod "232631ed-9f5b-4686-aae0-5ba0e6c202ee" (UID: "232631ed-9f5b-4686-aae0-5ba0e6c202ee"). InnerVolumeSpecName "kube-api-access-q95xq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:04.409480 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.409396    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q95xq\" (UniqueName: \"kubernetes.io/projected/232631ed-9f5b-4686-aae0-5ba0e6c202ee-kube-api-access-q95xq\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:04.409480 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.409439    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/232631ed-9f5b-4686-aae0-5ba0e6c202ee-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:04.409480 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.409454    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/232631ed-9f5b-4686-aae0-5ba0e6c202ee-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:04.429004 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.428955    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:50:04.429210 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.429175    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy" containerID="cri-o://d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef" gracePeriod=30
Jul 29 21:50:04.899116 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.899072    2565 generic.go:356] "Generic (PLEG): container finished" podID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" containerID="0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202" exitCode=0
Jul 29 21:50:04.899291 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.899136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"
Jul 29 21:50:04.899291 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.899131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" event={"ID":"232631ed-9f5b-4686-aae0-5ba0e6c202ee","Type":"ContainerDied","Data":"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"}
Jul 29 21:50:04.899291 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.899281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29" event={"ID":"232631ed-9f5b-4686-aae0-5ba0e6c202ee","Type":"ContainerDied","Data":"7ffe1426eaaad4229f8d2f7f25b2f4017afa2186b803d63712a1381e023c69e8"}
Jul 29 21:50:04.899450 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.899309    2565 scope.go:117] "RemoveContainer" containerID="0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"
Jul 29 21:50:04.909711 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.909690    2565 scope.go:117] "RemoveContainer" containerID="0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"
Jul 29 21:50:04.909999 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:04.909978    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202\": container with ID starting with 0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202 not found: ID does not exist" containerID="0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"
Jul 29 21:50:04.910126 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.910004    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202"} err="failed to get container status \"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202\": rpc error: code = NotFound desc = could not find container \"0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202\": container with ID starting with 0fc758e9925de8984fe5e3021c83c0f853533e582874ed9756a821149a6af202 not found: ID does not exist"
Jul 29 21:50:04.937209 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.937177    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:50:04.947622 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:04.947590    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-972150f2-ffd8fd578-vlj29"]
Jul 29 21:50:05.533454 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.533411    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:05.533856 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.533842    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" containerName="maas-api"
Jul 29 21:50:05.533901 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.533858    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" containerName="maas-api"
Jul 29 21:50:05.533936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.533923    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" containerName="maas-api"
Jul 29 21:50:05.538541 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.538519    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.541177 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.541149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9394560f-gw-openshift-default-dockercfg-6qcnx\""
Jul 29 21:50:05.555816 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.555779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:05.622145 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622322 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622368 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622448 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622484 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622463    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6pcl\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622536 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622536 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622614 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622560    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.622654 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.622617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6pcl\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724659 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724659 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724659 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724652    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.724819 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.724733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.725132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.725109    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.725251 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.725213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.725299 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.725248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.726890 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.726873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.726996 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.726975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.733248 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.733225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6pcl\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.733349 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.733274    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token\") pod \"e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") " pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:05.849914 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:05.849835    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:06.014157 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:50:06.014112    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af02ae3_e970_4f3d_b8bd_60beb635e75f.slice/crio-17cf526f279f8c20bea6db48e614134eede835147fb445fdab6e4b0c8e75b9e6 WatchSource:0}: Error finding container 17cf526f279f8c20bea6db48e614134eede835147fb445fdab6e4b0c8e75b9e6: Status 404 returned error can't find the container with id 17cf526f279f8c20bea6db48e614134eede835147fb445fdab6e4b0c8e75b9e6
Jul 29 21:50:06.014541 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.014293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:06.016730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.016688    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:06.016820 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.016785    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:06.016862 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.016830    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:06.055277 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.055245    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232631ed-9f5b-4686-aae0-5ba0e6c202ee" path="/var/lib/kubelet/pods/232631ed-9f5b-4686-aae0-5ba0e6c202ee/volumes"
Jul 29 21:50:06.913559 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.913523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerStarted","Data":"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"}
Jul 29 21:50:06.913726 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.913576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerStarted","Data":"17cf526f279f8c20bea6db48e614134eede835147fb445fdab6e4b0c8e75b9e6"}
Jul 29 21:50:06.937852 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:06.937781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podStartSLOduration=1.937759988 podStartE2EDuration="1.937759988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:06.937124123 +0000 UTC m=+2023.408703218" watchObservedRunningTime="2026-07-29 21:50:06.937759988 +0000 UTC m=+2023.409339065"
Jul 29 21:50:07.850704 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:07.850666    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:07.855508 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:07.855477    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:07.919281 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:07.919253    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:07.920272 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:07.920252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:08.618240 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.618202    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:08.622193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.622171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.626104 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.626073    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-9394560f-serving-cert\""
Jul 29 21:50:08.641672 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.641641    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:08.753221 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.753181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl5ls\" (UniqueName: \"kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.753421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.753311    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.753421 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.753353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.854609 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.854573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.854985 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.854618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.854985 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.854666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl5ls\" (UniqueName: \"kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.854985 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:08.854737    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-serving-cert: secret "maas-api-e2e-migrate-9394560f-serving-cert" not found
Jul 29 21:50:08.854985 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:08.854800    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls podName:00730bae-df61-413a-942e-a3a29e0fb252 nodeName:}" failed. No retries permitted until 2026-07-29 21:50:09.354783047 +0000 UTC m=+2025.826362101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls") pod "maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" (UID: "00730bae-df61-413a-942e-a3a29e0fb252") : secret "maas-api-e2e-migrate-9394560f-serving-cert" not found
Jul 29 21:50:08.855295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.855275    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:08.867347 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:08.867322    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl5ls\" (UniqueName: \"kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:09.360433 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.360391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:09.362983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.362956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") pod \"maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:09.533262 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.533229    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:09.690122 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.690091    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:50:09.695077 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.694872    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:09.765395 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d2b8\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765589 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765513    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765741 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765741 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765769    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.765851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765785    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:09.765851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:09.766008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765884    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket\") pod \"140e640e-9a7d-4995-adab-5c2b64bf3808\" (UID: \"140e640e-9a7d-4995-adab-5c2b64bf3808\") "
Jul 29 21:50:09.766008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.765892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:09.766198 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766112    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:09.766378 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766350    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e640e-9a7d-4995-adab-5c2b64bf3808-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.766378 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766376    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.766535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766388    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.766535 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766397    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.766731 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.766705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data" (OuterVolumeSpecName: "istio-data") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:09.768316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.768287    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token" (OuterVolumeSpecName: "istio-token") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:09.768316 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.768305    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8" (OuterVolumeSpecName: "kube-api-access-9d2b8") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "kube-api-access-9d2b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:09.768477 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.768383    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:50:09.768477 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.768397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "140e640e-9a7d-4995-adab-5c2b64bf3808" (UID: "140e640e-9a7d-4995-adab-5c2b64bf3808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:09.867841 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.867799    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.867841 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.867831    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.867841 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.867845    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.868308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.867857    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.868308 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.867869    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2b8\" (UniqueName: \"kubernetes.io/projected/140e640e-9a7d-4995-adab-5c2b64bf3808-kube-api-access-9d2b8\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:09.931673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.931632    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" event={"ID":"00730bae-df61-413a-942e-a3a29e0fb252","Type":"ContainerStarted","Data":"53601ebdd7b7dfd8429760dad8344a7424b2641c664fd273cfb98f009fee7369"}
Jul 29 21:50:09.933173 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.933144    2565 generic.go:356] "Generic (PLEG): container finished" podID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerID="d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef" exitCode=0
Jul 29 21:50:09.933285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.933206    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"
Jul 29 21:50:09.933285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.933219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerDied","Data":"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"}
Jul 29 21:50:09.933285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.933269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz" event={"ID":"140e640e-9a7d-4995-adab-5c2b64bf3808","Type":"ContainerDied","Data":"446928fa473d91a8e6b9b78a5364bc1eace5f19aef045efb1b59f7c73103deac"}
Jul 29 21:50:09.933388 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.933292    2565 scope.go:117] "RemoveContainer" containerID="d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"
Jul 29 21:50:09.943137 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.943115    2565 scope.go:117] "RemoveContainer" containerID="6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"
Jul 29 21:50:09.952028 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.952006    2565 scope.go:117] "RemoveContainer" containerID="d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"
Jul 29 21:50:09.952339 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:09.952321    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef\": container with ID starting with d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef not found: ID does not exist" containerID="d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"
Jul 29 21:50:09.952380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.952349    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef"} err="failed to get container status \"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef\": rpc error: code = NotFound desc = could not find container \"d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef\": container with ID starting with d64d62734a49f7561870349e395b5ca2c83532a95d647ab0dcd274c5eed548ef not found: ID does not exist"
Jul 29 21:50:09.952380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.952368    2565 scope.go:117] "RemoveContainer" containerID="6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"
Jul 29 21:50:09.952609 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:09.952595    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9\": container with ID starting with 6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9 not found: ID does not exist" containerID="6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"
Jul 29 21:50:09.952645 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.952613    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9"} err="failed to get container status \"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9\": rpc error: code = NotFound desc = could not find container \"6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9\": container with ID starting with 6ed7f1aec655caa8f27011865cbda09dc50717049cfe090242068b14ccd6ecf9 not found: ID does not exist"
Jul 29 21:50:09.976250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.976216    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:50:09.997345 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:09.997312    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-972150f2-openshift-default-74bcc99bfb-8ppmz"]
Jul 29 21:50:10.055764 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:10.055725    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" path="/var/lib/kubelet/pods/140e640e-9a7d-4995-adab-5c2b64bf3808/volumes"
Jul 29 21:50:10.940196 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:10.940160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" event={"ID":"00730bae-df61-413a-942e-a3a29e0fb252","Type":"ContainerStarted","Data":"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"}
Jul 29 21:50:10.940690 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:10.940227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:10.960649 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:10.960582    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" podStartSLOduration=2.400693251 podStartE2EDuration="2.960562988s" totalImagesPullingTime="559.869737ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:10.95849101 +0000 UTC m=+2027.430070087" watchObservedRunningTime="2026-07-29 21:50:10.960562988 +0000 UTC m=+2027.432142125"
Jul 29 21:50:16.961341 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:16.961303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:22.921321 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:22.921281    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:50:22.921815 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:22.921358    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 29 21:50:23.008386 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.008349    2565 generic.go:356] "Generic (PLEG): container finished" podID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerID="542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d" exitCode=0
Jul 29 21:50:23.008543 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.008417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerDied","Data":"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"}
Jul 29 21:50:23.008834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.008821    2565 scope.go:117] "RemoveContainer" containerID="542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"
Jul 29 21:50:23.009857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.009831    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:23.009930 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.009917    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:23.009967 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:23.009953    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:24.016404 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:24.016367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerStarted","Data":"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"}
Jul 29 21:50:24.850592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:24.850553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:24.850966 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:24.850941    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:50:24.851167 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:24.851139    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 29 21:50:25.850662 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:25.850589    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:50:25.851146 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:25.850687    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 29 21:50:26.850838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:26.850766    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:50:26.851376 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:26.850895    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 29 21:50:27.851250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:27.851211    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 29 21:50:27.851828 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:27.851796    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 29 21:50:28.854367 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:28.854336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:28.854772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:28.854579    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:28.855189 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:28.855173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:40.165682 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.165645    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:40.166267 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.165910    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" podUID="00730bae-df61-413a-942e-a3a29e0fb252" containerName="maas-api" containerID="cri-o://0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c" gracePeriod=30
Jul 29 21:50:40.431318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.431189    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:40.578350 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.578315    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") pod \"00730bae-df61-413a-942e-a3a29e0fb252\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") "
Jul 29 21:50:40.578537 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.578381    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl5ls\" (UniqueName: \"kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls\") pod \"00730bae-df61-413a-942e-a3a29e0fb252\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") "
Jul 29 21:50:40.578537 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.578459    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca\") pod \"00730bae-df61-413a-942e-a3a29e0fb252\" (UID: \"00730bae-df61-413a-942e-a3a29e0fb252\") "
Jul 29 21:50:40.579136 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.579089    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "00730bae-df61-413a-942e-a3a29e0fb252" (UID: "00730bae-df61-413a-942e-a3a29e0fb252"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:40.581362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.581330    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "00730bae-df61-413a-942e-a3a29e0fb252" (UID: "00730bae-df61-413a-942e-a3a29e0fb252"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:50:40.581517 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.581499    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls" (OuterVolumeSpecName: "kube-api-access-pl5ls") pod "00730bae-df61-413a-942e-a3a29e0fb252" (UID: "00730bae-df61-413a-942e-a3a29e0fb252"). InnerVolumeSpecName "kube-api-access-pl5ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:40.621540 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.621498    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:40.621769 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.621722    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" containerID="cri-o://18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632" gracePeriod=30
Jul 29 21:50:40.680027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.679988    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00730bae-df61-413a-942e-a3a29e0fb252-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:40.680027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.680022    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl5ls\" (UniqueName: \"kubernetes.io/projected/00730bae-df61-413a-942e-a3a29e0fb252-kube-api-access-pl5ls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:40.680027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:40.680035    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00730bae-df61-413a-942e-a3a29e0fb252-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:41.102696 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.102665    2565 generic.go:356] "Generic (PLEG): container finished" podID="00730bae-df61-413a-942e-a3a29e0fb252" containerID="0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c" exitCode=0
Jul 29 21:50:41.102861 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.102706    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"
Jul 29 21:50:41.102861 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.102718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" event={"ID":"00730bae-df61-413a-942e-a3a29e0fb252","Type":"ContainerDied","Data":"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"}
Jul 29 21:50:41.102861 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.102760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r" event={"ID":"00730bae-df61-413a-942e-a3a29e0fb252","Type":"ContainerDied","Data":"53601ebdd7b7dfd8429760dad8344a7424b2641c664fd273cfb98f009fee7369"}
Jul 29 21:50:41.102861 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.102781    2565 scope.go:117] "RemoveContainer" containerID="0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"
Jul 29 21:50:41.112086 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.112061    2565 scope.go:117] "RemoveContainer" containerID="0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"
Jul 29 21:50:41.112337 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:41.112318    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c\": container with ID starting with 0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c not found: ID does not exist" containerID="0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"
Jul 29 21:50:41.112399 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.112349    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c"} err="failed to get container status \"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c\": rpc error: code = NotFound desc = could not find container \"0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c\": container with ID starting with 0533a370028849ced9bd84f9b274ed12ebee9eee3908beb903f884d02e0bb95c not found: ID does not exist"
Jul 29 21:50:41.125027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.125002    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:41.130180 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.130156    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-9394560f-6c74c546b9-wrd9r"]
Jul 29 21:50:41.409987 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.409904    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411446    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00730bae-df61-413a-942e-a3a29e0fb252" containerName="maas-api"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411476    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="00730bae-df61-413a-942e-a3a29e0fb252" containerName="maas-api"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411496    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411508    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411689    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411702    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="00730bae-df61-413a-942e-a3a29e0fb252" containerName="maas-api"
Jul 29 21:50:41.415063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.411721    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="140e640e-9a7d-4995-adab-5c2b64bf3808" containerName="istio-proxy"
Jul 29 21:50:41.417686 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.417659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.420793 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.420771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0856dcaa-openshift-default-dockercfg-lhft4\""
Jul 29 21:50:41.447181 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.447153    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:50:41.488822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.488822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488824    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488841    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48fw9\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489037 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.488996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489254 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.489063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.489254 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.489089    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590709 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48fw9\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590881    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.590920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.590968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.591147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.591181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591273 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.591249    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591546 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.591394    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.591752 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.591730    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.593234 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.593203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.593417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.593395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.599576 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.599550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.600400 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.600356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48fw9\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9\") pod \"e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") " pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.729899 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.729861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:41.900328 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.900301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:50:41.902576 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:50:41.902545    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7138f7_6d9f_4de5_af94_12d87cd371ea.slice/crio-20086d777512ea42f6ca259cc210a07bea6465583a7d69e37ad5a6b93c7d225f WatchSource:0}: Error finding container 20086d777512ea42f6ca259cc210a07bea6465583a7d69e37ad5a6b93c7d225f: Status 404 returned error can't find the container with id 20086d777512ea42f6ca259cc210a07bea6465583a7d69e37ad5a6b93c7d225f
Jul 29 21:50:41.904837 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.904804    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:41.904914 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.904870    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:41.904914 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:41.904899    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:42.056965 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.056861    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00730bae-df61-413a-942e-a3a29e0fb252" path="/var/lib/kubelet/pods/00730bae-df61-413a-942e-a3a29e0fb252/volumes"
Jul 29 21:50:42.111845 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.111805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerStarted","Data":"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"}
Jul 29 21:50:42.112067 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.111863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerStarted","Data":"20086d777512ea42f6ca259cc210a07bea6465583a7d69e37ad5a6b93c7d225f"}
Jul 29 21:50:42.730881 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.730844    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:42.735961 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.735936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:42.759630 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:42.759565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podStartSLOduration=1.759546486 podStartE2EDuration="1.759546486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:42.145420741 +0000 UTC m=+2058.616999831" watchObservedRunningTime="2026-07-29 21:50:42.759546486 +0000 UTC m=+2059.231125561"
Jul 29 21:50:43.117208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:43.117120    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:43.118162 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:43.118144    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:50:43.857130 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:43.857093    2565 patch_prober.go:28] interesting pod/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:50:43.857539 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:43.857152    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:50:44.634320 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.634277    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:50:44.638444 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.638416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.641013 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.640986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0856dcaa-serving-cert\""
Jul 29 21:50:44.650406 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.650375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:50:44.721487 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.721446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.721658 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.721508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.721658 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.721576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.823027 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.822987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.823250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.823070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.823250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.823129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.823250 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:44.823182    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-serving-cert: secret "maas-api-e2e-ait-0856dcaa-serving-cert" not found
Jul 29 21:50:44.823407 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:44.823259    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls podName:b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6 nodeName:}" failed. No retries permitted until 2026-07-29 21:50:45.323238089 +0000 UTC m=+2061.794817156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls") pod "maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" (UID: "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6") : secret "maas-api-e2e-ait-0856dcaa-serving-cert" not found
Jul 29 21:50:44.823838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.823819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:44.832252 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:44.832226    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:45.328158 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:45.328126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:45.330492 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:45.330472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") pod \"maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:45.551962 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:45.551928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:45.748879 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:45.748756    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:50:45.751837 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:50:45.751811    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b35ef6_83e4_4b8b_ae9a_d5fa9af633a6.slice/crio-8a787dd93d1ef25de489a70a00d4fcb304fab1c021f8c56bb95deae98b7ab013 WatchSource:0}: Error finding container 8a787dd93d1ef25de489a70a00d4fcb304fab1c021f8c56bb95deae98b7ab013: Status 404 returned error can't find the container with id 8a787dd93d1ef25de489a70a00d4fcb304fab1c021f8c56bb95deae98b7ab013
Jul 29 21:50:45.897816 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:45.897789    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:46.039031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.038994    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039031 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039033    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039110    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039184    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039230    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6pcl\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039260    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039339    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:46.039754 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039352    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs\") pod \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\" (UID: \"3af02ae3-e970-4f3d-b8bd-60beb635e75f\") "
Jul 29 21:50:46.039754 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039724    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:50:46.039912 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.039851    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:46.040299 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040175    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:46.040676 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040643    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.040796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040750    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.040796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040767    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.040796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040781    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.040975 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.040799    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:46.041883 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.041845    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:50:46.041994 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.041937    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:46.042154 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.042129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl" (OuterVolumeSpecName: "kube-api-access-w6pcl") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "kube-api-access-w6pcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:50:46.042275 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.042207    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3af02ae3-e970-4f3d-b8bd-60beb635e75f" (UID: "3af02ae3-e970-4f3d-b8bd-60beb635e75f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:50:46.134512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.134412    2565 generic.go:356] "Generic (PLEG): container finished" podID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerID="18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632" exitCode=0
Jul 29 21:50:46.134512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.134459    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"
Jul 29 21:50:46.134512 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.134500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerDied","Data":"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"}
Jul 29 21:50:46.134777 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.134558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc" event={"ID":"3af02ae3-e970-4f3d-b8bd-60beb635e75f","Type":"ContainerDied","Data":"17cf526f279f8c20bea6db48e614134eede835147fb445fdab6e4b0c8e75b9e6"}
Jul 29 21:50:46.134777 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.134590    2565 scope.go:117] "RemoveContainer" containerID="18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"
Jul 29 21:50:46.135945 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.135918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" event={"ID":"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6","Type":"ContainerStarted","Data":"8a787dd93d1ef25de489a70a00d4fcb304fab1c021f8c56bb95deae98b7ab013"}
Jul 29 21:50:46.145095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.141602    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.145095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.141629    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.145095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.141644    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6pcl\" (UniqueName: \"kubernetes.io/projected/3af02ae3-e970-4f3d-b8bd-60beb635e75f-kube-api-access-w6pcl\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.145095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.141658    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.145095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.141673    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af02ae3-e970-4f3d-b8bd-60beb635e75f-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:50:46.145616 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.145589    2565 scope.go:117] "RemoveContainer" containerID="542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"
Jul 29 21:50:46.155391 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.155365    2565 scope.go:117] "RemoveContainer" containerID="18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"
Jul 29 21:50:46.155738 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:46.155716    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632\": container with ID starting with 18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632 not found: ID does not exist" containerID="18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"
Jul 29 21:50:46.155794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.155748    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632"} err="failed to get container status \"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632\": rpc error: code = NotFound desc = could not find container \"18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632\": container with ID starting with 18b0aa37ea313741924ff9207a8ad578c37a2d46c78060006b9109318cb80632 not found: ID does not exist"
Jul 29 21:50:46.155794 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.155769    2565 scope.go:117] "RemoveContainer" containerID="542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"
Jul 29 21:50:46.156101 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:50:46.156069    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d\": container with ID starting with 542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d not found: ID does not exist" containerID="542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"
Jul 29 21:50:46.156274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.156107    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d"} err="failed to get container status \"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d\": rpc error: code = NotFound desc = could not find container \"542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d\": container with ID starting with 542a0936746301455cf629123696745124084897455189b8b7c2eeaa188cd80d not found: ID does not exist"
Jul 29 21:50:46.157362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.157341    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:46.166358 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:46.166329    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9394560f-gw-openshift-default-6d79989b6d-qmhdc"]
Jul 29 21:50:47.141959 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:47.141924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" event={"ID":"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6","Type":"ContainerStarted","Data":"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"}
Jul 29 21:50:47.142495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:47.142008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:47.166776 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:47.166722    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" podStartSLOduration=2.631986347 podStartE2EDuration="3.166707841s" totalImagesPullingTime="534.721494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:47.164497613 +0000 UTC m=+2063.636076690" watchObservedRunningTime="2026-07-29 21:50:47.166707841 +0000 UTC m=+2063.638286916"
Jul 29 21:50:48.055325 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:48.055289    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" path="/var/lib/kubelet/pods/3af02ae3-e970-4f3d-b8bd-60beb635e75f/volumes"
Jul 29 21:50:53.165429 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:53.165394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:50:58.434555 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:58.433565    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:41314->10.132.0.51:15021: read: connection reset by peer" start-of-body=
Jul 29 21:50:58.434555 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:58.433639    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:41314->10.132.0.51:15021: read: connection reset by peer"
Jul 29 21:50:59.209477 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.209442    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerID="abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e" exitCode=0
Jul 29 21:50:59.209652 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.209507    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerDied","Data":"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"}
Jul 29 21:50:59.209925 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.209912    2565 scope.go:117] "RemoveContainer" containerID="abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"
Jul 29 21:50:59.211013 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.210988    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:59.211098 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.211076    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:50:59.211142 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:50:59.211109    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:00.216610 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:00.216578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerStarted","Data":"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"}
Jul 29 21:51:00.730441 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:00.730406    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:00.730613 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:00.730595    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:51:00.730658 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:00.730640    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 29 21:51:01.730374 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:01.730336    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:51:01.730763 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:01.730390    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 29 21:51:02.730716 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:02.730677    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:51:02.731172 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:02.730733    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 29 21:51:03.730873 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:03.730833    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 29 21:51:03.731378 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:03.730902    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 29 21:51:04.733911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:04.733871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:04.734662 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:04.734290    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:04.735019 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:04.734999    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:15.337270 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.337235    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:51:15.337792 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.337462    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" podUID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" containerName="maas-api" containerID="cri-o://a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6" gracePeriod=30
Jul 29 21:51:15.599549 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.599483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:51:15.744991 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.744956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") pod \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") "
Jul 29 21:51:15.745202 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.745034    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca\") pod \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") "
Jul 29 21:51:15.745202 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.745072    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788\") pod \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\" (UID: \"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6\") "
Jul 29 21:51:15.745445 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.745406    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" (UID: "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:51:15.747163 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.747133    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" (UID: "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:51:15.747259 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.747242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788" (OuterVolumeSpecName: "kube-api-access-m2788") pod "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" (UID: "b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6"). InnerVolumeSpecName "kube-api-access-m2788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:15.846285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.846252    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:15.846285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.846281    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:15.846285 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:15.846293    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2788\" (UniqueName: \"kubernetes.io/projected/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6-kube-api-access-m2788\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:16.299377 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.299338    2565 generic.go:356] "Generic (PLEG): container finished" podID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" containerID="a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6" exitCode=0
Jul 29 21:51:16.299534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.299392    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"
Jul 29 21:51:16.299534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.299419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" event={"ID":"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6","Type":"ContainerDied","Data":"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"}
Jul 29 21:51:16.299534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.299474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8" event={"ID":"b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6","Type":"ContainerDied","Data":"8a787dd93d1ef25de489a70a00d4fcb304fab1c021f8c56bb95deae98b7ab013"}
Jul 29 21:51:16.299534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.299499    2565 scope.go:117] "RemoveContainer" containerID="a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"
Jul 29 21:51:16.308253 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.308226    2565 scope.go:117] "RemoveContainer" containerID="a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"
Jul 29 21:51:16.308517 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:16.308498    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6\": container with ID starting with a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6 not found: ID does not exist" containerID="a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"
Jul 29 21:51:16.308576 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.308524    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6"} err="failed to get container status \"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6\": rpc error: code = NotFound desc = could not find container \"a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6\": container with ID starting with a6dbe39574ccfe09ceaede2c503c2ee0dea8dc46fc3b3b521cb8f445b65a16b6 not found: ID does not exist"
Jul 29 21:51:16.319743 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.319688    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:51:16.323558 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.323536    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0856dcaa-6548b545d-v2rn8"]
Jul 29 21:51:16.840362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.840323    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:51:16.840749 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:16.840609    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" containerID="cri-o://6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914" gracePeriod=30
Jul 29 21:51:17.681233 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681201    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:17.681624 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681610    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.681688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.681688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681650    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.681688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681656    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.681688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681672    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" containerName="maas-api"
Jul 29 21:51:17.681688 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681678    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" containerName="maas-api"
Jul 29 21:51:17.681837 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681735    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" containerName="maas-api"
Jul 29 21:51:17.681837 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681743    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.681837 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.681751    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af02ae3-e970-4f3d-b8bd-60beb635e75f" containerName="istio-proxy"
Jul 29 21:51:17.686611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.686590    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.691553 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.691531    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-48a37a48-openshift-default-dockercfg-9hd9s\""
Jul 29 21:51:17.703322 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.703297    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:17.765621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765621 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765661    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765790    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfkts\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.765842 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.766128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.766128 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.765949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867269 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867269 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfkts\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867703    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.867789 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867739    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.868314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867869    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.868314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.867996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.868314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.868099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.868314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.868303    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.869812 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.869792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.869905 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.869889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.875183 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.875161    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.875520 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.875503    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfkts\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts\") pod \"e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") " pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:17.996854 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:17.996763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:18.056947 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.056916    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6" path="/var/lib/kubelet/pods/b0b35ef6-83e4-4b8b-ae9a-d5fa9af633a6/volumes"
Jul 29 21:51:18.142631 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.142601    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:18.144869 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:51:18.144840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a582acc_75a6_458f_9ada_c992ecc70921.slice/crio-02f3f4fc96e291b8e0c1453f850bc9a6f8cc9215e3797d414a3adb24a328ed16 WatchSource:0}: Error finding container 02f3f4fc96e291b8e0c1453f850bc9a6f8cc9215e3797d414a3adb24a328ed16: Status 404 returned error can't find the container with id 02f3f4fc96e291b8e0c1453f850bc9a6f8cc9215e3797d414a3adb24a328ed16
Jul 29 21:51:18.147084 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.147056    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:18.147176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.147127    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:18.147176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.147158    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:18.313871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.313760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerStarted","Data":"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"}
Jul 29 21:51:18.313871 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.313826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerStarted","Data":"02f3f4fc96e291b8e0c1453f850bc9a6f8cc9215e3797d414a3adb24a328ed16"}
Jul 29 21:51:18.336656 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.336597    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podStartSLOduration=1.336581659 podStartE2EDuration="1.336581659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:18.333642501 +0000 UTC m=+2094.805221589" watchObservedRunningTime="2026-07-29 21:51:18.336581659 +0000 UTC m=+2094.808160735"
Jul 29 21:51:18.996858 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:18.996823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:19.736772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:19.736731    2565 patch_prober.go:28] interesting pod/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:51:19.736936 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:19.736791    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:51:20.002458 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.002373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:20.324984 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.324898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:20.326095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.326074    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:20.778697 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.778666    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:20.782643 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.782619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:20.784600 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.784578    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-48a37a48-serving-cert\""
Jul 29 21:51:20.793843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.793810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:20.899318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.899276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmzg\" (UniqueName: \"kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:20.899509 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.899319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:20.899509 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:20.899431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.000319 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.000285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmzg\" (UniqueName: \"kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.000319 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.000324    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.000577 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.000411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.001108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.001084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.002815 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.002796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.010527 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.010501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmzg\" (UniqueName: \"kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg\") pod \"maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.095488 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.095393    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:21.246892 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.246859    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:21.248614 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:51:21.248586    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d753f5_4c66_49d3_89e6_9a2874bd8405.slice/crio-dee566980a19bdcd99ee1ab2267327d0df6cdeb2d9ca1fbc24a0d718a6bccd54 WatchSource:0}: Error finding container dee566980a19bdcd99ee1ab2267327d0df6cdeb2d9ca1fbc24a0d718a6bccd54: Status 404 returned error can't find the container with id dee566980a19bdcd99ee1ab2267327d0df6cdeb2d9ca1fbc24a0d718a6bccd54
Jul 29 21:51:21.332018 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:21.331965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" event={"ID":"67d753f5-4c66-49d3-89e6-9a2874bd8405","Type":"ContainerStarted","Data":"dee566980a19bdcd99ee1ab2267327d0df6cdeb2d9ca1fbc24a0d718a6bccd54"}
Jul 29 21:51:22.110458 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.110404    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:22.212862 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.212829    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48fw9\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.212890    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.212932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213015 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.212971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213070    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213192    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo\") pod \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\" (UID: \"cf7138f7-6d9f-4de5-af94-12d87cd371ea\") "
Jul 29 21:51:22.213318 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213244    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:22.213635 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213388    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:51:22.213635 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213458    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:22.213796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213745    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.213796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213767    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.213796 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213783    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.213980 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.213957    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:22.214065 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.214006    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:22.215250 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.215225    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9" (OuterVolumeSpecName: "kube-api-access-48fw9") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "kube-api-access-48fw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:22.215424 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.215405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:22.215843 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.215824    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:51:22.215919 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.215835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf7138f7-6d9f-4de5-af94-12d87cd371ea" (UID: "cf7138f7-6d9f-4de5-af94-12d87cd371ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:22.315119 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315088    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.315119 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315114    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48fw9\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-kube-api-access-48fw9\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.315119 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315125    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.315366 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315134    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.315366 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315142    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.315366 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.315153    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7138f7-6d9f-4de5-af94-12d87cd371ea-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:22.339645 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.339608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" event={"ID":"67d753f5-4c66-49d3-89e6-9a2874bd8405","Type":"ContainerStarted","Data":"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"}
Jul 29 21:51:22.339812 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.339672    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:22.341388 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.341355    2565 generic.go:356] "Generic (PLEG): container finished" podID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerID="6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914" exitCode=0
Jul 29 21:51:22.341552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.341408    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"
Jul 29 21:51:22.341552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.341458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerDied","Data":"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"}
Jul 29 21:51:22.341552 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.341525    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8" event={"ID":"cf7138f7-6d9f-4de5-af94-12d87cd371ea","Type":"ContainerDied","Data":"20086d777512ea42f6ca259cc210a07bea6465583a7d69e37ad5a6b93c7d225f"}
Jul 29 21:51:22.341731 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.341559    2565 scope.go:117] "RemoveContainer" containerID="6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"
Jul 29 21:51:22.352461 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.352440    2565 scope.go:117] "RemoveContainer" containerID="abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"
Jul 29 21:51:22.357839 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.357779    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" podStartSLOduration=1.7052539599999998 podStartE2EDuration="2.357763111s" totalImagesPullingTime="652.509151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:22.356130602 +0000 UTC m=+2098.827709679" watchObservedRunningTime="2026-07-29 21:51:22.357763111 +0000 UTC m=+2098.829342187"
Jul 29 21:51:22.361720 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.361701    2565 scope.go:117] "RemoveContainer" containerID="6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"
Jul 29 21:51:22.362004 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:22.361983    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914\": container with ID starting with 6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914 not found: ID does not exist" containerID="6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"
Jul 29 21:51:22.362095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.362014    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914"} err="failed to get container status \"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914\": rpc error: code = NotFound desc = could not find container \"6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914\": container with ID starting with 6dad90f8b9b48d45a114b7377550b10bcbde090b8628af347eaa2679107f6914 not found: ID does not exist"
Jul 29 21:51:22.362095 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.362032    2565 scope.go:117] "RemoveContainer" containerID="abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"
Jul 29 21:51:22.362309 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:22.362290    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e\": container with ID starting with abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e not found: ID does not exist" containerID="abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"
Jul 29 21:51:22.362348 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.362315    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e"} err="failed to get container status \"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e\": rpc error: code = NotFound desc = could not find container \"abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e\": container with ID starting with abeebcb173a2bc2eb090fcc45ce2402f094345b0ffd1f8515553164f88ac224e not found: ID does not exist"
Jul 29 21:51:22.371584 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.371561    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:51:22.375207 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:22.375180    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0856dcaa-openshift-default-84b88584fb-sqqx8"]
Jul 29 21:51:24.055315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:24.055284    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" path="/var/lib/kubelet/pods/cf7138f7-6d9f-4de5-af94-12d87cd371ea/volumes"
Jul 29 21:51:28.365338 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:28.365298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:35.326335 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.326296    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:51:35.326746 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.326360    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 29 21:51:35.412616 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.412578    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a582acc-75a6-458f-9ada-c992ecc70921" containerID="4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b" exitCode=0
Jul 29 21:51:35.412788 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.412653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerDied","Data":"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"}
Jul 29 21:51:35.413132 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.413118    2565 scope.go:117] "RemoveContainer" containerID="4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"
Jul 29 21:51:35.414257 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.414231    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:35.414327 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.414297    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:35.414365 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:35.414328    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:51:36.419778 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:36.419742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerStarted","Data":"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"}
Jul 29 21:51:36.997507 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:36.997467    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:36.997715 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:36.997689    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:51:36.997800 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:36.997763    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 29 21:51:37.998204 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:37.998169    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:51:37.998608 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:37.998233    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 29 21:51:38.997033 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:38.996995    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:51:38.997217 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:38.997090    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 29 21:51:39.997557 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:39.997520    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 29 21:51:39.997949 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:39.997578    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 29 21:51:41.001523 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:41.001492    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:41.001863 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:41.001756    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:41.002515 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:41.002496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:51.453974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.453934    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:51.454601 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.454214    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" podUID="67d753f5-4c66-49d3-89e6-9a2874bd8405" containerName="maas-api" containerID="cri-o://f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768" gracePeriod=30
Jul 29 21:51:51.719684 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.719613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:51.813186 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.812685    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chmzg\" (UniqueName: \"kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg\") pod \"67d753f5-4c66-49d3-89e6-9a2874bd8405\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") "
Jul 29 21:51:51.813186 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.812748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls\") pod \"67d753f5-4c66-49d3-89e6-9a2874bd8405\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") "
Jul 29 21:51:51.813186 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.812859    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca\") pod \"67d753f5-4c66-49d3-89e6-9a2874bd8405\" (UID: \"67d753f5-4c66-49d3-89e6-9a2874bd8405\") "
Jul 29 21:51:51.813547 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.813516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "67d753f5-4c66-49d3-89e6-9a2874bd8405" (UID: "67d753f5-4c66-49d3-89e6-9a2874bd8405"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:51:51.816337 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.816307    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:51.816713 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.816539    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" containerID="cri-o://5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc" gracePeriod=30
Jul 29 21:51:51.816713 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.816581    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "67d753f5-4c66-49d3-89e6-9a2874bd8405" (UID: "67d753f5-4c66-49d3-89e6-9a2874bd8405"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:51:51.816713 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.816583    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg" (OuterVolumeSpecName: "kube-api-access-chmzg") pod "67d753f5-4c66-49d3-89e6-9a2874bd8405" (UID: "67d753f5-4c66-49d3-89e6-9a2874bd8405"). InnerVolumeSpecName "kube-api-access-chmzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:51.914324 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.914282    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67d753f5-4c66-49d3-89e6-9a2874bd8405-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:51.914324 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.914315    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmzg\" (UniqueName: \"kubernetes.io/projected/67d753f5-4c66-49d3-89e6-9a2874bd8405-kube-api-access-chmzg\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:51.914324 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:51.914326    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67d753f5-4c66-49d3-89e6-9a2874bd8405-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:52.500417 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.500386    2565 generic.go:356] "Generic (PLEG): container finished" podID="67d753f5-4c66-49d3-89e6-9a2874bd8405" containerID="f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768" exitCode=0
Jul 29 21:51:52.500810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.500431    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"
Jul 29 21:51:52.500810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.500479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" event={"ID":"67d753f5-4c66-49d3-89e6-9a2874bd8405","Type":"ContainerDied","Data":"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"}
Jul 29 21:51:52.500810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.500533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj" event={"ID":"67d753f5-4c66-49d3-89e6-9a2874bd8405","Type":"ContainerDied","Data":"dee566980a19bdcd99ee1ab2267327d0df6cdeb2d9ca1fbc24a0d718a6bccd54"}
Jul 29 21:51:52.500810 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.500558    2565 scope.go:117] "RemoveContainer" containerID="f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"
Jul 29 21:51:52.509574 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.509550    2565 scope.go:117] "RemoveContainer" containerID="f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"
Jul 29 21:51:52.509891 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:52.509869    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768\": container with ID starting with f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768 not found: ID does not exist" containerID="f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"
Jul 29 21:51:52.509942 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.509901    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768"} err="failed to get container status \"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768\": rpc error: code = NotFound desc = could not find container \"f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768\": container with ID starting with f117b7195cad1837c5ba6f9883557c45a58917c062aa2ac90ae9a9b5f0231768 not found: ID does not exist"
Jul 29 21:51:52.535673 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.535638    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:52.541538 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:52.541506    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-48a37a48-558bff8bc5-kvlcj"]
Jul 29 21:51:54.055011 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:54.054971    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67d753f5-4c66-49d3-89e6-9a2874bd8405" path="/var/lib/kubelet/pods/67d753f5-4c66-49d3-89e6-9a2874bd8405/volumes"
Jul 29 21:51:56.003844 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:56.003800    2565 patch_prober.go:28] interesting pod/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:51:56.004220 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:56.003882    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:51:57.094177 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.094148    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:57.271684 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271719    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271740    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271773    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271788    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfkts\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.271876 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.272303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.271933    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket\") pod \"6a582acc-75a6-458f-9ada-c992ecc70921\" (UID: \"6a582acc-75a6-458f-9ada-c992ecc70921\") "
Jul 29 21:51:57.272303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272067    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:57.272303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272273    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.272303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272296    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:57.272501 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272234    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:51:57.272674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data" (OuterVolumeSpecName: "istio-data") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:57.272674 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.272665    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:57.274108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.274075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:51:57.274108 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.274099    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token" (OuterVolumeSpecName: "istio-token") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:57.274248 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.274143    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:51:57.274303 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.274267    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts" (OuterVolumeSpecName: "kube-api-access-dfkts") pod "6a582acc-75a6-458f-9ada-c992ecc70921" (UID: "6a582acc-75a6-458f-9ada-c992ecc70921"). InnerVolumeSpecName "kube-api-access-dfkts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372742    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a582acc-75a6-458f-9ada-c992ecc70921-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372773    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfkts\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-kube-api-access-dfkts\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372784    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372797    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372808    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372816    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a582acc-75a6-458f-9ada-c992ecc70921-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372823    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a582acc-75a6-458f-9ada-c992ecc70921-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.372830 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.372830    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a582acc-75a6-458f-9ada-c992ecc70921-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:51:57.528091 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.528054    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a582acc-75a6-458f-9ada-c992ecc70921" containerID="5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc" exitCode=0
Jul 29 21:51:57.528239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.528119    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"
Jul 29 21:51:57.528239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.528143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerDied","Data":"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"}
Jul 29 21:51:57.528239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.528200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k" event={"ID":"6a582acc-75a6-458f-9ada-c992ecc70921","Type":"ContainerDied","Data":"02f3f4fc96e291b8e0c1453f850bc9a6f8cc9215e3797d414a3adb24a328ed16"}
Jul 29 21:51:57.528239 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.528222    2565 scope.go:117] "RemoveContainer" containerID="5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"
Jul 29 21:51:57.537419 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.537400    2565 scope.go:117] "RemoveContainer" containerID="4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"
Jul 29 21:51:57.545378 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.545361    2565 scope.go:117] "RemoveContainer" containerID="5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"
Jul 29 21:51:57.545663 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:57.545644    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc\": container with ID starting with 5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc not found: ID does not exist" containerID="5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"
Jul 29 21:51:57.545718 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.545672    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc"} err="failed to get container status \"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc\": rpc error: code = NotFound desc = could not find container \"5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc\": container with ID starting with 5964311ba97ef026a2db0575b23c4e578d299062795be246960c617bd70263fc not found: ID does not exist"
Jul 29 21:51:57.545718 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.545689    2565 scope.go:117] "RemoveContainer" containerID="4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"
Jul 29 21:51:57.545910 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:51:57.545894    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b\": container with ID starting with 4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b not found: ID does not exist" containerID="4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"
Jul 29 21:51:57.545953 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.545915    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b"} err="failed to get container status \"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b\": rpc error: code = NotFound desc = could not find container \"4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b\": container with ID starting with 4d942d0bb0db201cbae2f65e73e1e205dd8a9b10536f7aed4a2a292bf86cc25b not found: ID does not exist"
Jul 29 21:51:57.551383 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.551359    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:57.558012 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:57.557989    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-48a37a48-openshift-default-58784f84b8-tg46k"]
Jul 29 21:51:58.055570 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:51:58.055536    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" path="/var/lib/kubelet/pods/6a582acc-75a6-458f-9ada-c992ecc70921/volumes"
Jul 29 21:54:02.496380 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496292    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496692    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496704    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496716    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496728    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496734    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496751    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67d753f5-4c66-49d3-89e6-9a2874bd8405" containerName="maas-api"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496756    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="67d753f5-4c66-49d3-89e6-9a2874bd8405" containerName="maas-api"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496764    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496770    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496826    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496834    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496841    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="67d753f5-4c66-49d3-89e6-9a2874bd8405" containerName="maas-api"
Jul 29 21:54:02.496857 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.496848    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7138f7-6d9f-4de5-af94-12d87cd371ea" containerName="istio-proxy"
Jul 29 21:54:02.500156 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.500139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.504507 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.504483    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5dee3c8-openshift-default-dockercfg-hfxsm\""
Jul 29 21:54:02.526811 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.526774    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:54:02.615060 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615017    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615242 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615242 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzg6t\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615242 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615242 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615209    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615381 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615381 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615381 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.615381 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.615362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716641 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716641 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716647    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716737    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzg6t\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.716897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.716909    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.717170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717349 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.717245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717349 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.717338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717467 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.717444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.717550 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.717535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.718993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.718970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.719448 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.719431    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.727560 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.727535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.727791 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.727768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzg6t\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t\") pod \"e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") " pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.810863 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.810774    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:02.956750 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.956722    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:54:02.958831 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:54:02.958800    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b29a9af_5761_46a4_b95d_be6a8f9a38a9.slice/crio-32e78f17ffbacf1104324ec7638ee0c79ecda7bf2da3b0de602f7bcecbb9432e WatchSource:0}: Error finding container 32e78f17ffbacf1104324ec7638ee0c79ecda7bf2da3b0de602f7bcecbb9432e: Status 404 returned error can't find the container with id 32e78f17ffbacf1104324ec7638ee0c79ecda7bf2da3b0de602f7bcecbb9432e
Jul 29 21:54:02.961263 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.961229    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:02.961352 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.961292    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:02.961352 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:02.961345    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:03.155102 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:03.154981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerStarted","Data":"33afe5c392f64036229bb5fe78d49ba75f69108e4319df24e2fc33fbe6513080"}
Jul 29 21:54:03.155102 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:03.155032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerStarted","Data":"32e78f17ffbacf1104324ec7638ee0c79ecda7bf2da3b0de602f7bcecbb9432e"}
Jul 29 21:54:03.241534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:03.241477    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podStartSLOduration=1.241454553 podStartE2EDuration="1.241454553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:54:03.239729853 +0000 UTC m=+2259.711308929" watchObservedRunningTime="2026-07-29 21:54:03.241454553 +0000 UTC m=+2259.713033631"
Jul 29 21:54:03.811431 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:03.811398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:04.816657 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:04.816623    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:05.164798 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:05.164773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:05.165666 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:05.165643    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:06.578500 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.578470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:06.578952 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.578939    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a582acc-75a6-458f-9ada-c992ecc70921" containerName="istio-proxy"
Jul 29 21:54:06.582276 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.582257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.593827 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.593806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d5dee3c8-serving-cert\""
Jul 29 21:54:06.614510 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.614480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:06.649114 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.649079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.649293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.649118    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhc88\" (UniqueName: \"kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.649293 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.649154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.750616 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.750578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.750616 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.750620    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhc88\" (UniqueName: \"kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.750866 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.750641    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.750866 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:54:06.750779    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-serving-cert: secret "maas-api-e2e-mt-d5dee3c8-serving-cert" not found
Jul 29 21:54:06.750866 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:54:06.750861    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls podName:688338df-7818-4976-b4aa-17f2a972e84f nodeName:}" failed. No retries permitted until 2026-07-29 21:54:07.250844902 +0000 UTC m=+2263.722423956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls") pod "maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" (UID: "688338df-7818-4976-b4aa-17f2a972e84f") : secret "maas-api-e2e-mt-d5dee3c8-serving-cert" not found
Jul 29 21:54:06.751431 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.751402    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:06.762834 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:06.762805    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhc88\" (UniqueName: \"kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:07.256152 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:07.256105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:07.258488 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:07.258464    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") pod \"maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:07.492671 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:07.492638    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:07.642026 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:07.641999    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:07.644391 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:54:07.644350    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod688338df_7818_4976_b4aa_17f2a972e84f.slice/crio-7c4a9a6d11175ae4cdff19abd7e017211d8633729fe115b55134d8f999c44c8d WatchSource:0}: Error finding container 7c4a9a6d11175ae4cdff19abd7e017211d8633729fe115b55134d8f999c44c8d: Status 404 returned error can't find the container with id 7c4a9a6d11175ae4cdff19abd7e017211d8633729fe115b55134d8f999c44c8d
Jul 29 21:54:08.183343 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:08.183295    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" event={"ID":"688338df-7818-4976-b4aa-17f2a972e84f","Type":"ContainerStarted","Data":"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"}
Jul 29 21:54:08.183517 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:08.183393    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" event={"ID":"688338df-7818-4976-b4aa-17f2a972e84f","Type":"ContainerStarted","Data":"7c4a9a6d11175ae4cdff19abd7e017211d8633729fe115b55134d8f999c44c8d"}
Jul 29 21:54:08.183517 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:08.183426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:08.226962 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:08.226904    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" podStartSLOduration=1.8051998679999999 podStartE2EDuration="2.226888796s" totalImagesPullingTime="421.688928ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:54:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:54:08.225984052 +0000 UTC m=+2264.697563139" watchObservedRunningTime="2026-07-29 21:54:08.226888796 +0000 UTC m=+2264.698467871"
Jul 29 21:54:14.194712 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:14.194680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:20.728534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:20.727107    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:50972->10.132.0.55:15021: read: connection reset by peer" start-of-body=
Jul 29 21:54:20.728534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:20.727174    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": read tcp 10.132.0.2:50972->10.132.0.55:15021: read: connection reset by peer"
Jul 29 21:54:21.251877 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.251843    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerID="33afe5c392f64036229bb5fe78d49ba75f69108e4319df24e2fc33fbe6513080" exitCode=0
Jul 29 21:54:21.252061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.251913    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerDied","Data":"33afe5c392f64036229bb5fe78d49ba75f69108e4319df24e2fc33fbe6513080"}
Jul 29 21:54:21.252344 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.252330    2565 scope.go:117] "RemoveContainer" containerID="33afe5c392f64036229bb5fe78d49ba75f69108e4319df24e2fc33fbe6513080"
Jul 29 21:54:21.253392 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.253366    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:21.253492 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.253476    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:21.253534 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:21.253515    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:54:22.263732 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:22.263694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerStarted","Data":"16faca2fac3a759af77627bac4b0fcea2134169d78e404c7469cf4c213a2f135"}
Jul 29 21:54:22.811141 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:22.811106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:22.811434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:22.811409    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:54:22.811484 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:22.811467    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 21:54:23.811532 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:23.811493    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:54:23.811920 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:23.811557    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 21:54:24.811428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:24.811386    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:54:24.811610 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:24.811448    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 21:54:25.811596 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:25.811552    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 29 21:54:25.812100 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:25.811634    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 29 21:54:26.814989 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:26.814950    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:26.815518 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:26.815338    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:26.815911 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:26.815890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:54:48.404459 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.404426    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:48.405159 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.404684    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" podUID="688338df-7818-4976-b4aa-17f2a972e84f" containerName="maas-api" containerID="cri-o://56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb" gracePeriod=30
Jul 29 21:54:48.660153 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.660036    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:48.806935 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.806898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca\") pod \"688338df-7818-4976-b4aa-17f2a972e84f\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") "
Jul 29 21:54:48.807148 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.806954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhc88\" (UniqueName: \"kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88\") pod \"688338df-7818-4976-b4aa-17f2a972e84f\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") "
Jul 29 21:54:48.807148 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.806981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") pod \"688338df-7818-4976-b4aa-17f2a972e84f\" (UID: \"688338df-7818-4976-b4aa-17f2a972e84f\") "
Jul 29 21:54:48.807299 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.807275    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "688338df-7818-4976-b4aa-17f2a972e84f" (UID: "688338df-7818-4976-b4aa-17f2a972e84f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:54:48.807451 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.807434    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/688338df-7818-4976-b4aa-17f2a972e84f-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:54:48.809121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.809096    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88" (OuterVolumeSpecName: "kube-api-access-vhc88") pod "688338df-7818-4976-b4aa-17f2a972e84f" (UID: "688338df-7818-4976-b4aa-17f2a972e84f"). InnerVolumeSpecName "kube-api-access-vhc88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:54:48.809234 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.809091    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "688338df-7818-4976-b4aa-17f2a972e84f" (UID: "688338df-7818-4976-b4aa-17f2a972e84f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:54:48.908776 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.908730    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhc88\" (UniqueName: \"kubernetes.io/projected/688338df-7818-4976-b4aa-17f2a972e84f-kube-api-access-vhc88\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:54:48.908776 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:48.908764    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/688338df-7818-4976-b4aa-17f2a972e84f-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:54:49.403228 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.403194    2565 generic.go:356] "Generic (PLEG): container finished" podID="688338df-7818-4976-b4aa-17f2a972e84f" containerID="56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb" exitCode=0
Jul 29 21:54:49.403412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.403242    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"
Jul 29 21:54:49.403412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.403240    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" event={"ID":"688338df-7818-4976-b4aa-17f2a972e84f","Type":"ContainerDied","Data":"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"}
Jul 29 21:54:49.403412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.403359    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g" event={"ID":"688338df-7818-4976-b4aa-17f2a972e84f","Type":"ContainerDied","Data":"7c4a9a6d11175ae4cdff19abd7e017211d8633729fe115b55134d8f999c44c8d"}
Jul 29 21:54:49.403412 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.403381    2565 scope.go:117] "RemoveContainer" containerID="56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"
Jul 29 21:54:49.414452 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.414289    2565 scope.go:117] "RemoveContainer" containerID="56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"
Jul 29 21:54:49.414722 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:54:49.414635    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb\": container with ID starting with 56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb not found: ID does not exist" containerID="56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"
Jul 29 21:54:49.414722 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.414676    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb"} err="failed to get container status \"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb\": rpc error: code = NotFound desc = could not find container \"56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb\": container with ID starting with 56dd7583470d5718d79d8be07812e9291d5514db178881e5284f42c894f409fb not found: ID does not exist"
Jul 29 21:54:49.427648 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.427618    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:49.432697 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:49.432671    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5dee3c8-6456cdb8b6-hw46g"]
Jul 29 21:54:50.062101 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:50.062063    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="688338df-7818-4976-b4aa-17f2a972e84f" path="/var/lib/kubelet/pods/688338df-7818-4976-b4aa-17f2a972e84f/volumes"
Jul 29 21:54:55.252801 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:55.252764    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:54:55.253222 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:55.253011    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" containerID="cri-o://16faca2fac3a759af77627bac4b0fcea2134169d78e404c7469cf4c213a2f135" gracePeriod=30
Jul 29 21:54:56.817699 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:56.817659    2565 patch_prober.go:28] interesting pod/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 21:54:56.818126 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:54:56.817738    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 21:55:00.460885 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.460854    2565 generic.go:356] "Generic (PLEG): container finished" podID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerID="16faca2fac3a759af77627bac4b0fcea2134169d78e404c7469cf4c213a2f135" exitCode=0
Jul 29 21:55:00.461314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.460912    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerDied","Data":"16faca2fac3a759af77627bac4b0fcea2134169d78e404c7469cf4c213a2f135"}
Jul 29 21:55:00.461314 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.460960    2565 scope.go:117] "RemoveContainer" containerID="33afe5c392f64036229bb5fe78d49ba75f69108e4319df24e2fc33fbe6513080"
Jul 29 21:55:00.518191 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.518123    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:55:00.616893 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.616856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.616893 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.616895    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.616915    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.616940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.616966    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617016    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzg6t\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617093    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617161 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617145    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617206    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy\") pod \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\" (UID: \"3b29a9af-5761-46a4-b95d-be6a8f9a38a9\") "
Jul 29 21:55:00.617495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617325    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:55:00.617495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617399    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:55:00.617495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617425    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:55:00.617495 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617489    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:55:00.617772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617675    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.617772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617696    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.617772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617709    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.617772 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617723    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.617983 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.617811    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:55:00.619328 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.619304    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:55:00.619441 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.619405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t" (OuterVolumeSpecName: "kube-api-access-zzg6t") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "kube-api-access-zzg6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:55:00.619441 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.619419    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:55:00.619580 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.619498    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b29a9af-5761-46a4-b95d-be6a8f9a38a9" (UID: "3b29a9af-5761-46a4-b95d-be6a8f9a38a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:55:00.719034 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.718981    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.719034 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.719027    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.719034 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.719065    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzg6t\" (UniqueName: \"kubernetes.io/projected/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-kube-api-access-zzg6t\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.719315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.719081    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:00.719315 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:00.719095    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b29a9af-5761-46a4-b95d-be6a8f9a38a9-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:55:01.466893 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:01.466864    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"
Jul 29 21:55:01.466893 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:01.466883    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf" event={"ID":"3b29a9af-5761-46a4-b95d-be6a8f9a38a9","Type":"ContainerDied","Data":"32e78f17ffbacf1104324ec7638ee0c79ecda7bf2da3b0de602f7bcecbb9432e"}
Jul 29 21:55:01.467408 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:01.466935    2565 scope.go:117] "RemoveContainer" containerID="16faca2fac3a759af77627bac4b0fcea2134169d78e404c7469cf4c213a2f135"
Jul 29 21:55:01.489339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:01.489308    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:55:01.493518 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:01.493484    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5dee3c8-openshift-default-67b7cd8b48-hffpf"]
Jul 29 21:55:02.055218 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:55:02.055183    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" path="/var/lib/kubelet/pods/3b29a9af-5761-46a4-b95d-be6a8f9a38a9/volumes"
Jul 29 21:57:11.283827 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.283738    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:57:11.284376 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284354    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="688338df-7818-4976-b4aa-17f2a972e84f" containerName="maas-api"
Jul 29 21:57:11.284434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="688338df-7818-4976-b4aa-17f2a972e84f" containerName="maas-api"
Jul 29 21:57:11.284434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284402    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.284434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284412    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.284540 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284441    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.284540 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284450    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.284611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284548    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="688338df-7818-4976-b4aa-17f2a972e84f" containerName="maas-api"
Jul 29 21:57:11.284611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284560    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.284611 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.284570    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b29a9af-5761-46a4-b95d-be6a8f9a38a9" containerName="istio-proxy"
Jul 29 21:57:11.287828 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.287810    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.290274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.290252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9bee5593-openshift-default-dockercfg-96q5t\""
Jul 29 21:57:11.307974 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.307947    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:57:11.403028 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.402992    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403028 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nb8\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403295 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403263    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403469 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.403469 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.403362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504228 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504415 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504425    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nb8\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504592 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.504838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504707    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.505028 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.504994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.505171 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.505076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.505171 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.505090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.507208 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.507176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.507403 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.507387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.515258 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.515234    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.517138 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.517115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nb8\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8\") pod \"e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") " pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.598440 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.598347    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:11.739972 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.739945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:57:11.741698 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:57:11.741667    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca62c68_1fe6_4218_8648_71087a43d228.slice/crio-4423ed19a8aedc499ee4b480c07d3ae750e4762c1e3629aa5ce90b73270df420 WatchSource:0}: Error finding container 4423ed19a8aedc499ee4b480c07d3ae750e4762c1e3629aa5ce90b73270df420: Status 404 returned error can't find the container with id 4423ed19a8aedc499ee4b480c07d3ae750e4762c1e3629aa5ce90b73270df420
Jul 29 21:57:11.743918 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.743886    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:11.744014 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.743954    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:11.744014 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:11.744007    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:12.128123 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:12.128088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerStarted","Data":"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"}
Jul 29 21:57:12.128328 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:12.128144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerStarted","Data":"4423ed19a8aedc499ee4b480c07d3ae750e4762c1e3629aa5ce90b73270df420"}
Jul 29 21:57:12.151838 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:12.151781    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podStartSLOduration=1.151761101 podStartE2EDuration="1.151761101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:57:12.150617147 +0000 UTC m=+2448.622196224" watchObservedRunningTime="2026-07-29 21:57:12.151761101 +0000 UTC m=+2448.623340181"
Jul 29 21:57:12.598785 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:12.598750    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:12.603704 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:12.603679    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:13.132750 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:13.132719    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:13.133639 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:13.133620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:15.211198 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.211159    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:57:15.214824 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.214807    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.217225 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.217203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9bee5593-serving-cert\""
Jul 29 21:57:15.226456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.226431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:57:15.239846 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.239816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.240150 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.240125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.240362 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.240338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqg4n\" (UniqueName: \"kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.340993 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.340957    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.341205 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.340999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqg4n\" (UniqueName: \"kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.341205 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.341130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.341205 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:57:15.341140    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-serving-cert: secret "maas-api-e2e-mt-9bee5593-serving-cert" not found
Jul 29 21:57:15.341377 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:57:15.341215    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls podName:4ff67a78-df82-44f6-ae66-f00c0470844c nodeName:}" failed. No retries permitted until 2026-07-29 21:57:15.841194703 +0000 UTC m=+2452.312773777 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls") pod "maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" (UID: "4ff67a78-df82-44f6-ae66-f00c0470844c") : secret "maas-api-e2e-mt-9bee5593-serving-cert" not found
Jul 29 21:57:15.341822 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.341796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.349829 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.349804    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqg4n\" (UniqueName: \"kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.845740 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.845705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:15.848274 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:15.848248    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") pod \"maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:16.127215 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:16.127113    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:16.290762 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:16.290731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:57:16.292086 ip-10-0-129-123 kubenswrapper[2565]: W0729 21:57:16.292058    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff67a78_df82_44f6_ae66_f00c0470844c.slice/crio-0a3e1b9e297fc919c8bb53389167f55aab36683a2535ee67fc72be38849f4890 WatchSource:0}: Error finding container 0a3e1b9e297fc919c8bb53389167f55aab36683a2535ee67fc72be38849f4890: Status 404 returned error can't find the container with id 0a3e1b9e297fc919c8bb53389167f55aab36683a2535ee67fc72be38849f4890
Jul 29 21:57:16.293576 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:16.293555    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 21:57:17.157851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:17.157761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" event={"ID":"4ff67a78-df82-44f6-ae66-f00c0470844c","Type":"ContainerStarted","Data":"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"}
Jul 29 21:57:17.157851 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:17.157809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" event={"ID":"4ff67a78-df82-44f6-ae66-f00c0470844c","Type":"ContainerStarted","Data":"0a3e1b9e297fc919c8bb53389167f55aab36683a2535ee67fc72be38849f4890"}
Jul 29 21:57:17.158153 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:17.157893    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:17.178858 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:17.178799    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" podStartSLOduration=1.676369544 podStartE2EDuration="2.178777114s" totalImagesPullingTime="502.40757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:57:17.178105856 +0000 UTC m=+2453.649684932" watchObservedRunningTime="2026-07-29 21:57:17.178777114 +0000 UTC m=+2453.650356193"
Jul 29 21:57:23.169959 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:23.169927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:57:32.235089 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.235034    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca62c68-1fe6-4218-8648-71087a43d228" containerID="34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1" exitCode=0
Jul 29 21:57:32.235618 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.235110    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerDied","Data":"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"}
Jul 29 21:57:32.235618 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.235612    2565 scope.go:117] "RemoveContainer" containerID="34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"
Jul 29 21:57:32.236668 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.236639    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:32.236769 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.236720    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:32.236832 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:32.236766    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 21:57:33.246819 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:33.246778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerStarted","Data":"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"}
Jul 29 21:57:33.598817 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:33.598724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:33.599088 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:33.599027    2565 patch_prober.go:28] interesting pod/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 29 21:57:33.599139 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:33.599116    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 29 21:57:34.599498 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:34.599458    2565 patch_prober.go:28] interesting pod/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 29 21:57:34.599905 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:34.599528    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 29 21:57:35.598913 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:35.598880    2565 patch_prober.go:28] interesting pod/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 29 21:57:35.599121 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:35.598934    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 29 21:57:36.599323 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:36.599285    2565 patch_prober.go:28] interesting pod/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Jul 29 21:57:36.599732 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:36.599342    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Jul 29 21:57:37.603331 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:37.603294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:37.603805 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:37.603687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:57:37.604434 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:57:37.604418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:58:24.014414 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.014319    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:58:24.015011 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.014542    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" podUID="4ff67a78-df82-44f6-ae66-f00c0470844c" containerName="maas-api" containerID="cri-o://dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1" gracePeriod=30
Jul 29 21:58:24.287278 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.287253    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:58:24.349957 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.349925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") pod \"4ff67a78-df82-44f6-ae66-f00c0470844c\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") "
Jul 29 21:58:24.350176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.349988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca\") pod \"4ff67a78-df82-44f6-ae66-f00c0470844c\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") "
Jul 29 21:58:24.350176 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.350068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqg4n\" (UniqueName: \"kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n\") pod \"4ff67a78-df82-44f6-ae66-f00c0470844c\" (UID: \"4ff67a78-df82-44f6-ae66-f00c0470844c\") "
Jul 29 21:58:24.350396 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.350365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4ff67a78-df82-44f6-ae66-f00c0470844c" (UID: "4ff67a78-df82-44f6-ae66-f00c0470844c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:58:24.352119 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.352090    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4ff67a78-df82-44f6-ae66-f00c0470844c" (UID: "4ff67a78-df82-44f6-ae66-f00c0470844c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 21:58:24.352241 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.352181    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n" (OuterVolumeSpecName: "kube-api-access-wqg4n") pod "4ff67a78-df82-44f6-ae66-f00c0470844c" (UID: "4ff67a78-df82-44f6-ae66-f00c0470844c"). InnerVolumeSpecName "kube-api-access-wqg4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:58:24.451409 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.451372    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ff67a78-df82-44f6-ae66-f00c0470844c-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:58:24.451409 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.451408    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ff67a78-df82-44f6-ae66-f00c0470844c-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:58:24.451409 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.451420    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqg4n\" (UniqueName: \"kubernetes.io/projected/4ff67a78-df82-44f6-ae66-f00c0470844c-kube-api-access-wqg4n\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:58:24.531790 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.531699    2565 generic.go:356] "Generic (PLEG): container finished" podID="4ff67a78-df82-44f6-ae66-f00c0470844c" containerID="dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1" exitCode=0
Jul 29 21:58:24.531790 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.531761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" event={"ID":"4ff67a78-df82-44f6-ae66-f00c0470844c","Type":"ContainerDied","Data":"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"}
Jul 29 21:58:24.531790 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.531762    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"
Jul 29 21:58:24.532061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.531824    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd" event={"ID":"4ff67a78-df82-44f6-ae66-f00c0470844c","Type":"ContainerDied","Data":"0a3e1b9e297fc919c8bb53389167f55aab36683a2535ee67fc72be38849f4890"}
Jul 29 21:58:24.532061 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.531845    2565 scope.go:117] "RemoveContainer" containerID="dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"
Jul 29 21:58:24.542891 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.542868    2565 scope.go:117] "RemoveContainer" containerID="dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"
Jul 29 21:58:24.543194 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:58:24.543175    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1\": container with ID starting with dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1 not found: ID does not exist" containerID="dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"
Jul 29 21:58:24.543265 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.543202    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1"} err="failed to get container status \"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1\": rpc error: code = NotFound desc = could not find container \"dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1\": container with ID starting with dde72d8d5e86407416167c0d95873448c3265450ffdd8b50c53731687ef74ba1 not found: ID does not exist"
Jul 29 21:58:24.557428 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.557392    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:58:24.561730 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:24.561707    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9bee5593-7d6d5c5cd6-26lhd"]
Jul 29 21:58:26.055063 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:58:26.055022    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff67a78-df82-44f6-ae66-f00c0470844c" path="/var/lib/kubelet/pods/4ff67a78-df82-44f6-ae66-f00c0470844c/volumes"
Jul 29 21:59:16.439229 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:16.439192    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:59:16.439738 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:16.439414    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy" containerID="cri-o://1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546" gracePeriod=30
Jul 29 21:59:21.688505 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.688470    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:59:21.818456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818393    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818427    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818456 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818537    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818558    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818598    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9nb8\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8\") pod \"6ca62c68-1fe6-4218-8648-71087a43d228\" (UID: \"6ca62c68-1fe6-4218-8648-71087a43d228\") "
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818687    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:21.818872 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:21.819301 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.818978    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 21:59:21.819301 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819068    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.819301 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819088    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.819301 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819101    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.819301 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819114    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca62c68-1fe6-4218-8648-71087a43d228-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.819663 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819632    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:21.819821 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819801    2565 generic.go:356] "Generic (PLEG): container finished" podID="6ca62c68-1fe6-4218-8648-71087a43d228" containerID="1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546" exitCode=0
Jul 29 21:59:21.819897 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819880    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"
Jul 29 21:59:21.819945 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerDied","Data":"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"}
Jul 29 21:59:21.820008 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.819992    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj" event={"ID":"6ca62c68-1fe6-4218-8648-71087a43d228","Type":"ContainerDied","Data":"4423ed19a8aedc499ee4b480c07d3ae750e4762c1e3629aa5ce90b73270df420"}
Jul 29 21:59:21.820062 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.820031    2565 scope.go:117] "RemoveContainer" containerID="1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"
Jul 29 21:59:21.821100 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.821073    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8" (OuterVolumeSpecName: "kube-api-access-z9nb8") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "kube-api-access-z9nb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:59:21.821221 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.821192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 21:59:21.821367 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.821227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 21:59:21.821367 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.821317    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ca62c68-1fe6-4218-8648-71087a43d228" (UID: "6ca62c68-1fe6-4218-8648-71087a43d228"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 21:59:21.840339 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.840317    2565 scope.go:117] "RemoveContainer" containerID="34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"
Jul 29 21:59:21.848683 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.848662    2565 scope.go:117] "RemoveContainer" containerID="1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"
Jul 29 21:59:21.848970 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:59:21.848947    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546\": container with ID starting with 1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546 not found: ID does not exist" containerID="1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"
Jul 29 21:59:21.849029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.848979    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546"} err="failed to get container status \"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546\": rpc error: code = NotFound desc = could not find container \"1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546\": container with ID starting with 1a1885249d7f91023a377532a660b270b791de22209b42f9cbfbbd4900a19546 not found: ID does not exist"
Jul 29 21:59:21.849029 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.849000    2565 scope.go:117] "RemoveContainer" containerID="34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"
Jul 29 21:59:21.849254 ip-10-0-129-123 kubenswrapper[2565]: E0729 21:59:21.849236    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1\": container with ID starting with 34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1 not found: ID does not exist" containerID="34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"
Jul 29 21:59:21.849310 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.849260    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1"} err="failed to get container status \"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1\": rpc error: code = NotFound desc = could not find container \"34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1\": container with ID starting with 34d21ee05b99438331e504d16629ed03f56b5550838641ba1be9080e539ee8a1 not found: ID does not exist"
Jul 29 21:59:21.920193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.920150    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca62c68-1fe6-4218-8648-71087a43d228-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.920193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.920185    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.920193 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.920198    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nb8\" (UniqueName: \"kubernetes.io/projected/6ca62c68-1fe6-4218-8648-71087a43d228-kube-api-access-z9nb8\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.920422 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.920209    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:21.920422 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:21.920221    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca62c68-1fe6-4218-8648-71087a43d228-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 21:59:22.139307 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:22.139225    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:59:22.144090 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:22.144065    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9bee5593-openshift-default-7d44dd4f5b-9nlvj"]
Jul 29 21:59:24.054865 ip-10-0-129-123 kubenswrapper[2565]: I0729 21:59:24.054835    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" path="/var/lib/kubelet/pods/6ca62c68-1fe6-4218-8648-71087a43d228/volumes"
Jul 29 22:00:00.135512 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135424    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"]
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135842    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135853    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135862    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff67a78-df82-44f6-ae66-f00c0470844c" containerName="maas-api"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135868    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff67a78-df82-44f6-ae66-f00c0470844c" containerName="maas-api"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135880    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135887    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135951    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135960    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff67a78-df82-44f6-ae66-f00c0470844c" containerName="maas-api"
Jul 29 22:00:00.135985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.135966    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca62c68-1fe6-4218-8648-71087a43d228" containerName="istio-proxy"
Jul 29 22:00:00.139401 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.139381    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"
Jul 29 22:00:00.142343 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.142320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\""
Jul 29 22:00:00.148170 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.148141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"]
Jul 29 22:00:00.150181 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.150161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"
Jul 29 22:00:00.292739 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:00.292711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"]
Jul 29 22:00:00.295204 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:00:00.295178    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode626741b_e008_4b88_96a1_d888543f418a.slice/crio-4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1 WatchSource:0}: Error finding container 4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1: Status 404 returned error can't find the container with id 4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1
Jul 29 22:00:01.021483 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:01.021449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt" event={"ID":"e626741b-e008-4b88-96a1-d888543f418a","Type":"ContainerStarted","Data":"256fcdf127c03cc67d4c623c9ffdd884af59034281ffad929ddace38e5255343"}
Jul 29 22:00:01.021682 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:01.021512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt" event={"ID":"e626741b-e008-4b88-96a1-d888543f418a","Type":"ContainerStarted","Data":"4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1"}
Jul 29 22:00:01.038653 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:01.038599    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt" podStartSLOduration=1.038580544 podStartE2EDuration="1.038580544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:01.037499887 +0000 UTC m=+2617.509078945" watchObservedRunningTime="2026-07-29 22:00:01.038580544 +0000 UTC m=+2617.510159622"
Jul 29 22:00:02.027742 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:02.027706    2565 generic.go:356] "Generic (PLEG): container finished" podID="e626741b-e008-4b88-96a1-d888543f418a" containerID="256fcdf127c03cc67d4c623c9ffdd884af59034281ffad929ddace38e5255343" exitCode=0
Jul 29 22:00:02.028202 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:02.027792    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt" event={"ID":"e626741b-e008-4b88-96a1-d888543f418a","Type":"ContainerDied","Data":"256fcdf127c03cc67d4c623c9ffdd884af59034281ffad929ddace38e5255343"}
Jul 29 22:00:03.165681 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:03.165652    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"
Jul 29 22:00:04.039330 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:04.039300    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt"
Jul 29 22:00:04.039330 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:04.039310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29756040-rc6bt" event={"ID":"e626741b-e008-4b88-96a1-d888543f418a","Type":"ContainerDied","Data":"4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1"}
Jul 29 22:00:04.039531 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:04.039352    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c83791e763151cde0339e9e1040e49eb9debb87692f18899cefd5b36bd924d1"
Jul 29 22:00:35.258811 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.258780    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:00:35.259284 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.259195    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e626741b-e008-4b88-96a1-d888543f418a" containerName="cleanup"
Jul 29 22:00:35.259284 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.259207    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e626741b-e008-4b88-96a1-d888543f418a" containerName="cleanup"
Jul 29 22:00:35.259284 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.259283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e626741b-e008-4b88-96a1-d888543f418a" containerName="cleanup"
Jul 29 22:00:35.262351 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.262334    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.266671 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.266651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-25c9e8-openshift-default-dockercfg-m9hbs\""
Jul 29 22:00:35.292225 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.292190    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:00:35.337767 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337720    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.337959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.337959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.337959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.337959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.338243 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.337977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stsqs\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.338243 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.338091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.338243 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.338140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.338376 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.338246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439292 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439292 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stsqs\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439900 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439974 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439904    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439974 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.439974 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439962    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.440155 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.439873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.440351 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.440333    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.441840 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.441814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.441955 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.441871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.449747 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.449712    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.450028 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.450003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stsqs\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs\") pod \"e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") " pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.572554 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.572474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:35.720546 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.720375    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:00:35.723515 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:00:35.723489    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fdd0b4_aecc_4eb2_827b_0b3afcd1d4c4.slice/crio-76c8a0edc5c791908cfc3a48de34c58386c2391d4f60671d1dca714fee63e28b WatchSource:0}: Error finding container 76c8a0edc5c791908cfc3a48de34c58386c2391d4f60671d1dca714fee63e28b: Status 404 returned error can't find the container with id 76c8a0edc5c791908cfc3a48de34c58386c2391d4f60671d1dca714fee63e28b
Jul 29 22:00:35.726027 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.725997    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:35.726106 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.726081    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:35.726155 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:35.726110    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:36.197832 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:36.197803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerStarted","Data":"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"}
Jul 29 22:00:36.198069 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:36.197864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerStarted","Data":"76c8a0edc5c791908cfc3a48de34c58386c2391d4f60671d1dca714fee63e28b"}
Jul 29 22:00:36.228581 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:36.228532    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podStartSLOduration=1.228516509 podStartE2EDuration="1.228516509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:36.228125797 +0000 UTC m=+2652.699704874" watchObservedRunningTime="2026-07-29 22:00:36.228516509 +0000 UTC m=+2652.700095586"
Jul 29 22:00:36.573274 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:36.573192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:36.577834 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:36.577810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:37.203391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:37.203363    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:37.204270 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:37.204249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:39.516536 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.516492    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:00:39.520420 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.520397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.523875 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:00:39.523838    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.524476 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:00:39.524453    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-inf-a-25c9e8-serving-cert\" is forbidden: User \"system:node:ip-10-0-129-123.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-123.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-25c9e8-serving-cert\"" type="*v1.Secret"
Jul 29 22:00:39.568393 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.568354    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:00:39.687422 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.687379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.687607 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.687450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.687607 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.687509    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdzv4\" (UniqueName: \"kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.788830 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.788730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.788830 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.788789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.789094 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.788902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdzv4\" (UniqueName: \"kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.789472 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.789453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:39.799700 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:39.799679    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdzv4\" (UniqueName: \"kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:40.506567 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:40.506533    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-25c9e8-serving-cert\""
Jul 29 22:00:40.511613 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:40.511584    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls\") pod \"maas-api-e2e-inf-a-25c9e8-6996784787-2chxk\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:40.731528 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:40.731489    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:40.875017 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:40.874988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:00:40.876478 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:00:40.876453    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e09f7b_74ef_4330_adf8_62fcc7fd6af1.slice/crio-7aa04748a8796eaa5849576719149da775995f688be6306f083647708bf69b70 WatchSource:0}: Error finding container 7aa04748a8796eaa5849576719149da775995f688be6306f083647708bf69b70: Status 404 returned error can't find the container with id 7aa04748a8796eaa5849576719149da775995f688be6306f083647708bf69b70
Jul 29 22:00:41.224532 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:41.224492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" event={"ID":"29e09f7b-74ef-4330-adf8-62fcc7fd6af1","Type":"ContainerStarted","Data":"7aa04748a8796eaa5849576719149da775995f688be6306f083647708bf69b70"}
Jul 29 22:00:42.230818 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:42.230778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" event={"ID":"29e09f7b-74ef-4330-adf8-62fcc7fd6af1","Type":"ContainerStarted","Data":"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"}
Jul 29 22:00:42.231272 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:42.230855    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:42.252479 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:42.252425    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" podStartSLOduration=2.078062804 podStartE2EDuration="3.252406904s" totalImagesPullingTime="1.1743441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:00:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:00:42.250233936 +0000 UTC m=+2658.721813023" watchObservedRunningTime="2026-07-29 22:00:42.252406904 +0000 UTC m=+2658.723985980"
Jul 29 22:00:48.254454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:48.254422    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:00:52.205414 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.205374    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 29 22:00:52.205846 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.205443    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 22:00:52.309296 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.309264    2565 generic.go:356] "Generic (PLEG): container finished" podID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerID="5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5" exitCode=0
Jul 29 22:00:52.309466 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.309335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerDied","Data":"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"}
Jul 29 22:00:52.309775 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.309760    2565 scope.go:117] "RemoveContainer" containerID="5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"
Jul 29 22:00:52.310984 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.310956    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:52.311081 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.311038    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:52.311133 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:52.311100    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:00:53.317741 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:53.317700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerStarted","Data":"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"}
Jul 29 22:00:53.573704 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:53.573595    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:53.573959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:53.573854    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 29 22:00:53.573959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:53.573915    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 22:00:54.573641 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:54.573606    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 29 22:00:54.574036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:54.573663    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 22:00:55.573666 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:55.573621    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 29 22:00:55.574111 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:55.573684    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 22:00:56.573977 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:56.573933    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Jul 29 22:00:56.574391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:56.574003    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Jul 29 22:00:57.578074 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:57.578024    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:57.578493 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:57.578405    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:00:57.579338 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:00:57.579315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:01:04.082426 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.082395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:01:04.085967 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.085948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.090504 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.090485    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c24a76-openshift-default-dockercfg-lzwts\""
Jul 29 22:01:04.142958 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.142923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:01:04.157245 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157218    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157388 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157388 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157495 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157542 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157592 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtg22\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157669 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157728 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.157779 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.157735    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259336 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259336 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259552 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259402    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259552 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259649 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtg22\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259649 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259861 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259861 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.259959 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.260093 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.259965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.260174 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.260152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.260234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.260210    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.260342 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.260325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.262001 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.261973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.262114 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.262020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.272401 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.272380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtg22\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.274292 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.274271    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token\") pod \"e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") " pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.395901 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.395817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:04.580366 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.580225    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:01:04.582526 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:01:04.582503    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d431829_81e0_4fce_936a_76ee0bda10c7.slice/crio-79b78dd461cbe1aafdf52b04ba72237c860666d172b3fe5da068fec9c5da7e5d WatchSource:0}: Error finding container 79b78dd461cbe1aafdf52b04ba72237c860666d172b3fe5da068fec9c5da7e5d: Status 404 returned error can't find the container with id 79b78dd461cbe1aafdf52b04ba72237c860666d172b3fe5da068fec9c5da7e5d
Jul 29 22:01:04.584735 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.584704    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:04.584825 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.584769    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:04.584825 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:04.584795    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:05.382454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:05.382407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerStarted","Data":"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"}
Jul 29 22:01:05.382857 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:05.382472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerStarted","Data":"79b78dd461cbe1aafdf52b04ba72237c860666d172b3fe5da068fec9c5da7e5d"}
Jul 29 22:01:05.395994 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:05.395969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:05.400803 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:05.400781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:05.409612 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:05.409569    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podStartSLOduration=1.409554723 podStartE2EDuration="1.409554723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:01:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:01:05.407785739 +0000 UTC m=+2681.879364828" watchObservedRunningTime="2026-07-29 22:01:05.409554723 +0000 UTC m=+2681.881133798"
Jul 29 22:01:06.387970 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:06.387938    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:06.389038 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:06.389014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:08.717990 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.717952    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:01:08.722011 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.721990    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.724574 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.724552    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c24a76-serving-cert\""
Jul 29 22:01:08.734700 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.734675    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:01:08.806063 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.806013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.806253 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.806112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.806253 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.806165    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsmz\" (UniqueName: \"kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.906821 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.906786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.906999 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.906838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.906999 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.906889    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsmz\" (UniqueName: \"kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.907551 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.907527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.909216 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.909197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:08.916920 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:08.916898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsmz\" (UniqueName: \"kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz\") pod \"maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:09.034673 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:09.034577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:09.186385 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:09.186355    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:01:09.187567 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:01:09.187540    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecf0100_ff8d_4563_9af2_63280981eabc.slice/crio-3e946ca025bb0e1c9ddda7e3f7fb5b52364ae3d448211dbb53b5acef797c0559 WatchSource:0}: Error finding container 3e946ca025bb0e1c9ddda7e3f7fb5b52364ae3d448211dbb53b5acef797c0559: Status 404 returned error can't find the container with id 3e946ca025bb0e1c9ddda7e3f7fb5b52364ae3d448211dbb53b5acef797c0559
Jul 29 22:01:09.405006 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:09.404907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" event={"ID":"2ecf0100-ff8d-4563-9af2-63280981eabc","Type":"ContainerStarted","Data":"3e946ca025bb0e1c9ddda7e3f7fb5b52364ae3d448211dbb53b5acef797c0559"}
Jul 29 22:01:10.411833 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:10.411793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" event={"ID":"2ecf0100-ff8d-4563-9af2-63280981eabc","Type":"ContainerStarted","Data":"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"}
Jul 29 22:01:10.412295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:10.411916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:10.434217 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:10.434148    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" podStartSLOduration=2.017888273 podStartE2EDuration="2.4341259s" totalImagesPullingTime="416.237627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:01:10.431859683 +0000 UTC m=+2686.903438762" watchObservedRunningTime="2026-07-29 22:01:10.4341259 +0000 UTC m=+2686.905704977"
Jul 29 22:01:16.423596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:16.423508    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:01:22.430241 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:22.430201    2565 patch_prober.go:28] interesting pod/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 22:01:22.430619 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:22.430274    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 22:01:23.492391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.492318    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerID="fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886" exitCode=0
Jul 29 22:01:23.492774 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.492384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerDied","Data":"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"}
Jul 29 22:01:23.492830 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.492816    2565 scope.go:117] "RemoveContainer" containerID="fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"
Jul 29 22:01:23.493854 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.493829    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:23.493939 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.493893    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:23.493939 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:23.493925    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:01:24.499662 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:24.499615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerStarted","Data":"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"}
Jul 29 22:01:25.396227 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:25.396183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:25.396481 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:25.396454    2565 patch_prober.go:28] interesting pod/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 29 22:01:25.396567 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:25.396518    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 29 22:01:26.396351 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:26.396314    2565 patch_prober.go:28] interesting pod/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 29 22:01:26.396763 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:26.396368    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 29 22:01:27.396663 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:27.396630    2565 patch_prober.go:28] interesting pod/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 29 22:01:27.397036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:27.396698    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 29 22:01:28.400458 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:28.400426    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:28.400914 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:28.400735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:01:28.401549 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:01:28.401530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:03:04.280801 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.280697    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:03:04.281376 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.280939    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" containerName="maas-api" containerID="cri-o://e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685" gracePeriod=30
Jul 29 22:03:04.540703 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.540628    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:03:04.596371 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.596019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdzv4\" (UniqueName: \"kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4\") pod \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") "
Jul 29 22:03:04.596371 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.596143    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca\") pod \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") "
Jul 29 22:03:04.596371 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.596236    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls\") pod \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\" (UID: \"29e09f7b-74ef-4330-adf8-62fcc7fd6af1\") "
Jul 29 22:03:04.597169 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.596817    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "29e09f7b-74ef-4330-adf8-62fcc7fd6af1" (UID: "29e09f7b-74ef-4330-adf8-62fcc7fd6af1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:04.597169 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.596965    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:04.601101 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.601026    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "29e09f7b-74ef-4330-adf8-62fcc7fd6af1" (UID: "29e09f7b-74ef-4330-adf8-62fcc7fd6af1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:04.604331 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.604266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4" (OuterVolumeSpecName: "kube-api-access-gdzv4") pod "29e09f7b-74ef-4330-adf8-62fcc7fd6af1" (UID: "29e09f7b-74ef-4330-adf8-62fcc7fd6af1"). InnerVolumeSpecName "kube-api-access-gdzv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:04.697627 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.697592    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdzv4\" (UniqueName: \"kubernetes.io/projected/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-kube-api-access-gdzv4\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:04.697627 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:04.697620    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29e09f7b-74ef-4330-adf8-62fcc7fd6af1-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:05.018793 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.018755    2565 generic.go:356] "Generic (PLEG): container finished" podID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" containerID="e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685" exitCode=0
Jul 29 22:03:05.018989 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.018805    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"
Jul 29 22:03:05.018989 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.018837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" event={"ID":"29e09f7b-74ef-4330-adf8-62fcc7fd6af1","Type":"ContainerDied","Data":"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"}
Jul 29 22:03:05.018989 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.018888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk" event={"ID":"29e09f7b-74ef-4330-adf8-62fcc7fd6af1","Type":"ContainerDied","Data":"7aa04748a8796eaa5849576719149da775995f688be6306f083647708bf69b70"}
Jul 29 22:03:05.018989 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.018910    2565 scope.go:117] "RemoveContainer" containerID="e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"
Jul 29 22:03:05.028589 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.028570    2565 scope.go:117] "RemoveContainer" containerID="e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"
Jul 29 22:03:05.028853 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:05.028832    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685\": container with ID starting with e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685 not found: ID does not exist" containerID="e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"
Jul 29 22:03:05.028950 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.028861    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685"} err="failed to get container status \"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685\": rpc error: code = NotFound desc = could not find container \"e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685\": container with ID starting with e5eda69a89adacb2e2a35fe39f57e42853cdfcf87201a66e298b59dda0fdf685 not found: ID does not exist"
Jul 29 22:03:05.041186 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.041159    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:03:05.045824 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:05.045800    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-25c9e8-6996784787-2chxk"]
Jul 29 22:03:06.063536 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:06.063501    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" path="/var/lib/kubelet/pods/29e09f7b-74ef-4330-adf8-62fcc7fd6af1/volumes"
Jul 29 22:03:11.819651 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:11.819616    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:03:11.820186 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:11.819851    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" containerID="cri-o://3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c" gracePeriod=30
Jul 29 22:03:12.580192 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:12.580156    2565 patch_prober.go:28] interesting pod/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 29 22:03:12.580362 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:12.580214    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 29 22:03:17.084122 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.084058    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:03:17.084455 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.084416    2565 generic.go:356] "Generic (PLEG): container finished" podID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerID="3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c" exitCode=0
Jul 29 22:03:17.084501 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.084483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerDied","Data":"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"}
Jul 29 22:03:17.084554 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.084541    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm" event={"ID":"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4","Type":"ContainerDied","Data":"76c8a0edc5c791908cfc3a48de34c58386c2391d4f60671d1dca714fee63e28b"}
Jul 29 22:03:17.084589 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.084567    2565 scope.go:117] "RemoveContainer" containerID="3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"
Jul 29 22:03:17.092979 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.092962    2565 scope.go:117] "RemoveContainer" containerID="5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"
Jul 29 22:03:17.101359 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.101329    2565 scope.go:117] "RemoveContainer" containerID="3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"
Jul 29 22:03:17.101683 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:17.101664    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c\": container with ID starting with 3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c not found: ID does not exist" containerID="3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"
Jul 29 22:03:17.101767 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.101697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c"} err="failed to get container status \"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c\": rpc error: code = NotFound desc = could not find container \"3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c\": container with ID starting with 3279b845b6e74f74f96db0b2ca773465a4b92e95e8723bfadba24ec60dd7ea6c not found: ID does not exist"
Jul 29 22:03:17.101767 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.101723    2565 scope.go:117] "RemoveContainer" containerID="5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"
Jul 29 22:03:17.101995 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:17.101979    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5\": container with ID starting with 5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5 not found: ID does not exist" containerID="5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"
Jul 29 22:03:17.102066 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.102004    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5"} err="failed to get container status \"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5\": rpc error: code = NotFound desc = could not find container \"5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5\": container with ID starting with 5674cbb1f96902f358ba6c5bd45ccfa0afdde5f512a752e4dfc1c2f6865181e5 not found: ID does not exist"
Jul 29 22:03:17.228969 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.228937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229154 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229154 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229062    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stsqs\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229154 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229154 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229154 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229137    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229430 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229430 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229430 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:17.229590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:17.229590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229462    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:17.229590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229475    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data\") pod \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\" (UID: \"91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4\") "
Jul 29 22:03:17.229590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229560    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:17.229940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229917    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.230031 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229946    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.230031 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229959    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.230031 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.229973    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.230285 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.230258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:17.231415 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.231388    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs" (OuterVolumeSpecName: "kube-api-access-stsqs") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "kube-api-access-stsqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:17.231501 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.231421    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:17.231501 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.231453    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:17.231501 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.231461    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" (UID: "91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:03:17.330575 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.330546    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stsqs\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-kube-api-access-stsqs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.330575 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.330572    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.330575 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.330581    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.330766 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.330589    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:17.330766 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:17.330599    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:18.090889 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:18.090860    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"
Jul 29 22:03:18.113439 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:18.113406    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:03:18.116889 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:18.116859    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-25c9e8-openshift-default-586d4c4f58-zjbmm"]
Jul 29 22:03:20.055054 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:20.055009    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" path="/var/lib/kubelet/pods/91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4/volumes"
Jul 29 22:03:22.808378 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:22.808336    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:03:22.808848 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:22.808608    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" podUID="2ecf0100-ff8d-4563-9af2-63280981eabc" containerName="maas-api" containerID="cri-o://d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8" gracePeriod=30
Jul 29 22:03:23.069881 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.069814    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:03:23.119566 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.119534    2565 generic.go:356] "Generic (PLEG): container finished" podID="2ecf0100-ff8d-4563-9af2-63280981eabc" containerID="d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8" exitCode=0
Jul 29 22:03:23.119771 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.119586    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"
Jul 29 22:03:23.119771 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.119588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" event={"ID":"2ecf0100-ff8d-4563-9af2-63280981eabc","Type":"ContainerDied","Data":"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"}
Jul 29 22:03:23.119771 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.119741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct" event={"ID":"2ecf0100-ff8d-4563-9af2-63280981eabc","Type":"ContainerDied","Data":"3e946ca025bb0e1c9ddda7e3f7fb5b52364ae3d448211dbb53b5acef797c0559"}
Jul 29 22:03:23.119925 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.119788    2565 scope.go:117] "RemoveContainer" containerID="d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"
Jul 29 22:03:23.128744 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.128722    2565 scope.go:117] "RemoveContainer" containerID="d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"
Jul 29 22:03:23.129098 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:23.129026    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8\": container with ID starting with d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8 not found: ID does not exist" containerID="d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"
Jul 29 22:03:23.129098 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.129080    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8"} err="failed to get container status \"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8\": rpc error: code = NotFound desc = could not find container \"d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8\": container with ID starting with d8a2c9cefe8510b11e9b1fee3567bce7c1be806882e0ee35f7c30e270b31ceb8 not found: ID does not exist"
Jul 29 22:03:23.192713 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.192677    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca\") pod \"2ecf0100-ff8d-4563-9af2-63280981eabc\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") "
Jul 29 22:03:23.192927 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.192733    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls\") pod \"2ecf0100-ff8d-4563-9af2-63280981eabc\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") "
Jul 29 22:03:23.192927 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.192778    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwsmz\" (UniqueName: \"kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz\") pod \"2ecf0100-ff8d-4563-9af2-63280981eabc\" (UID: \"2ecf0100-ff8d-4563-9af2-63280981eabc\") "
Jul 29 22:03:23.193167 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.193132    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ecf0100-ff8d-4563-9af2-63280981eabc" (UID: "2ecf0100-ff8d-4563-9af2-63280981eabc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:23.195132 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.195089    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ecf0100-ff8d-4563-9af2-63280981eabc" (UID: "2ecf0100-ff8d-4563-9af2-63280981eabc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:03:23.195258 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.195232    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz" (OuterVolumeSpecName: "kube-api-access-hwsmz") pod "2ecf0100-ff8d-4563-9af2-63280981eabc" (UID: "2ecf0100-ff8d-4563-9af2-63280981eabc"). InnerVolumeSpecName "kube-api-access-hwsmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:23.293805 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.293757    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ecf0100-ff8d-4563-9af2-63280981eabc-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:23.293805 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.293801    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ecf0100-ff8d-4563-9af2-63280981eabc-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:23.293805 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.293814    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsmz\" (UniqueName: \"kubernetes.io/projected/2ecf0100-ff8d-4563-9af2-63280981eabc-kube-api-access-hwsmz\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:23.446951 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.446914    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:03:23.451360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:23.451329    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c24a76-78fb8946ff-84kct"]
Jul 29 22:03:24.055274 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:24.055235    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ecf0100-ff8d-4563-9af2-63280981eabc" path="/var/lib/kubelet/pods/2ecf0100-ff8d-4563-9af2-63280981eabc/volumes"
Jul 29 22:03:30.198518 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:30.198472    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:03:30.198988 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:30.198686    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy" containerID="cri-o://e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9" gracePeriod=30
Jul 29 22:03:31.175351 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175318    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:03:31.175748 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175735    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:31.175795 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175750    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:31.175795 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ecf0100-ff8d-4563-9af2-63280981eabc" containerName="maas-api"
Jul 29 22:03:31.175795 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175774    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ecf0100-ff8d-4563-9af2-63280981eabc" containerName="maas-api"
Jul 29 22:03:31.175795 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175787    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" containerName="maas-api"
Jul 29 22:03:31.175795 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" containerName="maas-api"
Jul 29 22:03:31.175940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175859    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:31.175940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175869    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e09f7b-74ef-4330-adf8-62fcc7fd6af1" containerName="maas-api"
Jul 29 22:03:31.175940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175875    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:31.175940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.175884    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ecf0100-ff8d-4563-9af2-63280981eabc" containerName="maas-api"
Jul 29 22:03:31.180637 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.180614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.183258 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.183233    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c8e5f0-openshift-default-dockercfg-djlns\""
Jul 29 22:03:31.197679 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.197646    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:03:31.277982 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.277945    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.277990    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndc4\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.278463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.278327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379143 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379143 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379356 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379272    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379356 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379356 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379499 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379552 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379552 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379657 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndc4\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379657 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379597    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379657 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379815 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379815 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.379926 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.379895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.381799 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.381775    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.381973 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.381951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.388884 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.388860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.389320 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.389289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndc4\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4\") pod \"e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") " pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.491000 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.490908    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:31.651585 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.651557    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:03:31.653561 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:03:31.653532    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c8e04b_e7ff_4f86_a6e6_0b53ed4f887f.slice/crio-5e416e9cf3d71deac63979550e27ffe9a3b47ebb4207a2664e4298f7bc2d005d WatchSource:0}: Error finding container 5e416e9cf3d71deac63979550e27ffe9a3b47ebb4207a2664e4298f7bc2d005d: Status 404 returned error can't find the container with id 5e416e9cf3d71deac63979550e27ffe9a3b47ebb4207a2664e4298f7bc2d005d
Jul 29 22:03:31.655588 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.655554    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:31.655706 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.655624    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:31.655706 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:31.655653    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:32.177843 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:32.177803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerStarted","Data":"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"}
Jul 29 22:03:32.178034 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:32.177858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerStarted","Data":"5e416e9cf3d71deac63979550e27ffe9a3b47ebb4207a2664e4298f7bc2d005d"}
Jul 29 22:03:32.203762 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:32.203694    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podStartSLOduration=1.203674715 podStartE2EDuration="1.203674715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:03:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:32.203444736 +0000 UTC m=+2828.675023814" watchObservedRunningTime="2026-07-29 22:03:32.203674715 +0000 UTC m=+2828.675253792"
Jul 29 22:03:32.492018 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:32.491929    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:32.497523 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:32.497493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:33.184109 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:33.184078    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:33.185213 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:33.185191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:35.489098 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.489073    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.518824    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtg22\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.518922    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.518979    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519031    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519148    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519182    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.520113 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.519269    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token\") pod \"3d431829-81e0-4fce-936a-76ee0bda10c7\" (UID: \"3d431829-81e0-4fce-936a-76ee0bda10c7\") "
Jul 29 22:03:35.521972 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.521940    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:35.522359 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.522335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:35.522886 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.522830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22" (OuterVolumeSpecName: "kube-api-access-xtg22") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "kube-api-access-xtg22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:35.523633 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.523392    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:03:35.525377 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.524427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:35.526385 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.526358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:35.526955 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.526932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:03:35.527289 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.527266    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:03:35.529011 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.528989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d431829-81e0-4fce-936a-76ee0bda10c7" (UID: "3d431829-81e0-4fce-936a-76ee0bda10c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:03:35.621205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621172    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtg22\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-kube-api-access-xtg22\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621198    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621208    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621216    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621225    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621233    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621240    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d431829-81e0-4fce-936a-76ee0bda10c7-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621248    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d431829-81e0-4fce-936a-76ee0bda10c7-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.621427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.621256    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d431829-81e0-4fce-936a-76ee0bda10c7-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:03:35.644145 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644108    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:03:35.644640 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644620    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:35.644640 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644640    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="91fdd0b4-aecc-4eb2-827b-0b3afcd1d4c4" containerName="istio-proxy"
Jul 29 22:03:35.644837 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644658    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:03:35.644837 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644664    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:03:35.644837 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:03:35.644837 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644683    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:03:35.644837 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.644792    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:03:35.648316 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.648295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.652391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.652368    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c8e5f0-serving-cert\""
Jul 29 22:03:35.667813 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.667783    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:03:35.722445 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.722354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.722590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.722489    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.722590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.722532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khfw\" (UniqueName: \"kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.823725 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.823682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.823905 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.823811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.823905 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:35.823844    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-serving-cert: secret "maas-api-e2e-api-a-c8e5f0-serving-cert" not found
Jul 29 22:03:35.824009 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:35.823927    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls podName:408c162c-51d3-46c8-8cac-7117f2c2624e nodeName:}" failed. No retries permitted until 2026-07-29 22:03:36.323905124 +0000 UTC m=+2832.795484179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls") pod "maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" (UID: "408c162c-51d3-46c8-8cac-7117f2c2624e") : secret "maas-api-e2e-api-a-c8e5f0-serving-cert" not found
Jul 29 22:03:35.824009 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.823852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khfw\" (UniqueName: \"kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.824502 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.824481    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:35.840387 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:35.840362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khfw\" (UniqueName: \"kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:36.200683 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.200646    2565 generic.go:356] "Generic (PLEG): container finished" podID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerID="e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9" exitCode=0
Jul 29 22:03:36.200832 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.200722    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"
Jul 29 22:03:36.200832 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.200738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerDied","Data":"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"}
Jul 29 22:03:36.200832 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.200802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5" event={"ID":"3d431829-81e0-4fce-936a-76ee0bda10c7","Type":"ContainerDied","Data":"79b78dd461cbe1aafdf52b04ba72237c860666d172b3fe5da068fec9c5da7e5d"}
Jul 29 22:03:36.200832 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.200828    2565 scope.go:117] "RemoveContainer" containerID="e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"
Jul 29 22:03:36.210024 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.210005    2565 scope.go:117] "RemoveContainer" containerID="fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"
Jul 29 22:03:36.226770 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.226744    2565 scope.go:117] "RemoveContainer" containerID="e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"
Jul 29 22:03:36.227240 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:36.227203    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9\": container with ID starting with e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9 not found: ID does not exist" containerID="e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"
Jul 29 22:03:36.227372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.227251    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9"} err="failed to get container status \"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9\": rpc error: code = NotFound desc = could not find container \"e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9\": container with ID starting with e5d8e6e80ad0b0486482696bc9ecc4587e3f68be8fae8e4c5c968a5eeb3ec1c9 not found: ID does not exist"
Jul 29 22:03:36.227372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.227276    2565 scope.go:117] "RemoveContainer" containerID="fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"
Jul 29 22:03:36.227557 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:03:36.227538    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886\": container with ID starting with fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886 not found: ID does not exist" containerID="fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"
Jul 29 22:03:36.227613 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.227569    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886"} err="failed to get container status \"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886\": rpc error: code = NotFound desc = could not find container \"fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886\": container with ID starting with fc83f851f6dd7aad356e481f15f73932d255b097ac912ffd60c746ba35e73886 not found: ID does not exist"
Jul 29 22:03:36.241215 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.241183    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:03:36.262039 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.262012    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c24a76-openshift-default-57c9c64458-rlkx5"]
Jul 29 22:03:36.329001 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.328908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:36.331425 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.331399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") pod \"maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:36.560641 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.560611    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:36.919929 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:03:36.919899    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408c162c_51d3_46c8_8cac_7117f2c2624e.slice/crio-93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a WatchSource:0}: Error finding container 93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a: Status 404 returned error can't find the container with id 93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a
Jul 29 22:03:36.920400 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.920369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:03:36.921143 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:36.921126    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:03:37.207211 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:37.207120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" event={"ID":"408c162c-51d3-46c8-8cac-7117f2c2624e","Type":"ContainerStarted","Data":"93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a"}
Jul 29 22:03:38.055919 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:38.055887    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" path="/var/lib/kubelet/pods/3d431829-81e0-4fce-936a-76ee0bda10c7/volumes"
Jul 29 22:03:38.214940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:38.214905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" event={"ID":"408c162c-51d3-46c8-8cac-7117f2c2624e","Type":"ContainerStarted","Data":"bc0f1e88f86e12100d0c30d61f8facb9c465616229cecbfbca307a12baa72a20"}
Jul 29 22:03:38.215121 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:38.214969    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:38.261346 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:38.261291    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" podStartSLOduration=2.134795589 podStartE2EDuration="3.261273756s" totalImagesPullingTime="1.126478167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:03:38.2565724 +0000 UTC m=+2834.728151477" watchObservedRunningTime="2026-07-29 22:03:38.261273756 +0000 UTC m=+2834.732852831"
Jul 29 22:03:44.225810 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:44.225781    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:03:48.938359 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:48.937433    2565 patch_prober.go:28] interesting pod/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:60064->10.132.0.64:15021: read: connection reset by peer" start-of-body=
Jul 29 22:03:48.938359 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:48.937499    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": read tcp 10.132.0.2:60064->10.132.0.64:15021: read: connection reset by peer"
Jul 29 22:03:49.300062 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.299953    2565 generic.go:356] "Generic (PLEG): container finished" podID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerID="41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48" exitCode=0
Jul 29 22:03:49.300062 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.299999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerDied","Data":"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"}
Jul 29 22:03:49.300443 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.300428    2565 scope.go:117] "RemoveContainer" containerID="41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"
Jul 29 22:03:49.301470 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.301442    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:49.301528 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.301514    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:49.301564 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:49.301547    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:03:50.306977 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:50.306942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerStarted","Data":"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"}
Jul 29 22:03:50.491511 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:50.491465    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:50.491762 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:50.491736    2565 patch_prober.go:28] interesting pod/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 29 22:03:50.491831 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:50.491789    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 29 22:03:51.491610 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:51.491566    2565 patch_prober.go:28] interesting pod/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 29 22:03:51.492007 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:51.491641    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 29 22:03:52.491985 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:52.491950    2565 patch_prober.go:28] interesting pod/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 29 22:03:52.492419 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:52.492030    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 29 22:03:53.491971 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:53.491931    2565 patch_prober.go:28] interesting pod/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 29 22:03:53.492184 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:53.491992    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 29 22:03:54.495261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:54.495227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:54.495767 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:54.495629    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:03:54.496473 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:03:54.496451    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:04:05.338220 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.338183    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:04:05.338669 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.338656    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d431829-81e0-4fce-936a-76ee0bda10c7" containerName="istio-proxy"
Jul 29 22:04:05.341808 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.341785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.348265 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.348242    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ab119-openshift-default-dockercfg-jwpk2\""
Jul 29 22:04:05.385506 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.385463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:04:05.424537 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424728 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424728 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424930 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424930 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.424930 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5tvh\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.425088 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.424957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525718 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525718 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525745    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5tvh\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525840    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.525975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.525915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.526384 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.526291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.526384 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.526326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.526495 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.526404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.526611 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.526585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.526681 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.526625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.528277 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.528255    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.528467 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.528451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.537680 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.537660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5tvh\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.537777 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.537710    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token\") pod \"e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") " pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.654610 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.654509    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:05.819446 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.819416    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:04:05.821494 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:04:05.821461    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e4c86b_a16c_49f6_b845_fa7f50807bda.slice/crio-482c33fda113dd251d29b5762d806b4eb2fb70f6748478b31e970e8219b311aa WatchSource:0}: Error finding container 482c33fda113dd251d29b5762d806b4eb2fb70f6748478b31e970e8219b311aa: Status 404 returned error can't find the container with id 482c33fda113dd251d29b5762d806b4eb2fb70f6748478b31e970e8219b311aa
Jul 29 22:04:05.823692 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.823663    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:05.823782 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.823728    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:05.823782 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:05.823756    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:06.401975 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:06.401936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerStarted","Data":"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"}
Jul 29 22:04:06.402460 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:06.401997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerStarted","Data":"482c33fda113dd251d29b5762d806b4eb2fb70f6748478b31e970e8219b311aa"}
Jul 29 22:04:06.429012 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:06.428953    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podStartSLOduration=1.428937698 podStartE2EDuration="1.428937698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:04:06.427089483 +0000 UTC m=+2862.898668558" watchObservedRunningTime="2026-07-29 22:04:06.428937698 +0000 UTC m=+2862.900516774"
Jul 29 22:04:06.655728 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:06.655652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:06.660530 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:06.660505    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:07.407620 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:07.407591    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:07.408801 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:07.408777    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:09.640898 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.640863    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:04:09.644793 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.644768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.647239 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.647214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-6ab119-serving-cert\""
Jul 29 22:04:09.659243 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.659215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:04:09.778906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.778866    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.778906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.778913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.779174 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.778931    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6cs\" (UniqueName: \"kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.880408 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.880363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.880601 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.880419    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.880601 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:04:09.880517    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-serving-cert: secret "maas-api-e2e-api-b-6ab119-serving-cert" not found
Jul 29 22:04:09.880601 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.880541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6cs\" (UniqueName: \"kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.880601 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:04:09.880589    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls podName:a4256062-b3fa-4d01-8f09-e42ec3501b2e nodeName:}" failed. No retries permitted until 2026-07-29 22:04:10.380566978 +0000 UTC m=+2866.852146032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls") pod "maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" (UID: "a4256062-b3fa-4d01-8f09-e42ec3501b2e") : secret "maas-api-e2e-api-b-6ab119-serving-cert" not found
Jul 29 22:04:09.881213 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.881192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:09.893538 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:09.893458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6cs\" (UniqueName: \"kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:10.386664 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:10.386628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:10.389213 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:10.389191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") pod \"maas-api-e2e-api-b-6ab119-678b8c7f47-542j2\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:10.555888 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:10.555849    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:10.724470 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:10.724409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:04:10.727779 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:04:10.727743    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4256062_b3fa_4d01_8f09_e42ec3501b2e.slice/crio-40c505c515c8238c0f5075f190e9c498e6e76d415c121a214238421d2200ed92 WatchSource:0}: Error finding container 40c505c515c8238c0f5075f190e9c498e6e76d415c121a214238421d2200ed92: Status 404 returned error can't find the container with id 40c505c515c8238c0f5075f190e9c498e6e76d415c121a214238421d2200ed92
Jul 29 22:04:11.430941 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:11.430906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" event={"ID":"a4256062-b3fa-4d01-8f09-e42ec3501b2e","Type":"ContainerStarted","Data":"40c505c515c8238c0f5075f190e9c498e6e76d415c121a214238421d2200ed92"}
Jul 29 22:04:12.438155 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:12.438106    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" event={"ID":"a4256062-b3fa-4d01-8f09-e42ec3501b2e","Type":"ContainerStarted","Data":"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"}
Jul 29 22:04:12.438541 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:12.438197    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:12.458199 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:12.458147    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" podStartSLOduration=2.336927677 podStartE2EDuration="3.458130808s" totalImagesPullingTime="1.121203131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:04:12.456576377 +0000 UTC m=+2868.928155448" watchObservedRunningTime="2026-07-29 22:04:12.458130808 +0000 UTC m=+2868.929709883"
Jul 29 22:04:18.454610 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:18.454503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:04:23.264001 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.263304    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:36348->10.132.0.66:15021: read: connection reset by peer" start-of-body=
Jul 29 22:04:23.264001 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.263388    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:36348->10.132.0.66:15021: read: connection reset by peer"
Jul 29 22:04:23.507572 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.507484    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerID="c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30" exitCode=0
Jul 29 22:04:23.507732 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.507562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerDied","Data":"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"}
Jul 29 22:04:23.508106 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.508090    2565 scope.go:117] "RemoveContainer" containerID="c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"
Jul 29 22:04:23.509269 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.509239    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:23.509402 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.509322    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:23.509402 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:23.509379    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:04:24.515309 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:24.515274    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerStarted","Data":"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"}
Jul 29 22:04:24.655281 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:24.655188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:24.655491 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:24.655463    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 29 22:04:24.655570 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:24.655525    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 29 22:04:25.656650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:25.656608    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 29 22:04:25.657150 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:25.656673    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 29 22:04:26.655563 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:26.655523    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 29 22:04:26.655778 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:26.655587    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 29 22:04:27.655546 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:27.655496    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 29 22:04:27.656070 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:27.655586    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 29 22:04:28.655373 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:28.655328    2565 patch_prober.go:28] interesting pod/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 29 22:04:28.655755 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:28.655687    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 29 22:04:29.659602 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:29.659524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:29.659955 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:29.659896    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:29.660610 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:29.660593    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:04:52.429109 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.429071    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:04:52.429716 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.429362    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" podUID="408c162c-51d3-46c8-8cac-7117f2c2624e" containerName="maas-api" containerID="cri-o://bc0f1e88f86e12100d0c30d61f8facb9c465616229cecbfbca307a12baa72a20" gracePeriod=30
Jul 29 22:04:52.667585 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.667445    2565 generic.go:356] "Generic (PLEG): container finished" podID="408c162c-51d3-46c8-8cac-7117f2c2624e" containerID="bc0f1e88f86e12100d0c30d61f8facb9c465616229cecbfbca307a12baa72a20" exitCode=0
Jul 29 22:04:52.667585 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.667523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" event={"ID":"408c162c-51d3-46c8-8cac-7117f2c2624e","Type":"ContainerDied","Data":"bc0f1e88f86e12100d0c30d61f8facb9c465616229cecbfbca307a12baa72a20"}
Jul 29 22:04:52.667716 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.667587    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44" event={"ID":"408c162c-51d3-46c8-8cac-7117f2c2624e","Type":"ContainerDied","Data":"93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a"}
Jul 29 22:04:52.667716 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.667606    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93decc70e18355c60f31e06b217b3724891bdd717e9a68ee4226cc439ec96d3a"
Jul 29 22:04:52.688638 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.688551    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:04:52.725879 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.725847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca\") pod \"408c162c-51d3-46c8-8cac-7117f2c2624e\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") "
Jul 29 22:04:52.726147 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.725938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") pod \"408c162c-51d3-46c8-8cac-7117f2c2624e\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") "
Jul 29 22:04:52.726147 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.726030    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7khfw\" (UniqueName: \"kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw\") pod \"408c162c-51d3-46c8-8cac-7117f2c2624e\" (UID: \"408c162c-51d3-46c8-8cac-7117f2c2624e\") "
Jul 29 22:04:52.726343 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.726299    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "408c162c-51d3-46c8-8cac-7117f2c2624e" (UID: "408c162c-51d3-46c8-8cac-7117f2c2624e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:04:52.726542 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.726510    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/408c162c-51d3-46c8-8cac-7117f2c2624e-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:04:52.728632 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.728602    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "408c162c-51d3-46c8-8cac-7117f2c2624e" (UID: "408c162c-51d3-46c8-8cac-7117f2c2624e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:04:52.728725 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.728622    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw" (OuterVolumeSpecName: "kube-api-access-7khfw") pod "408c162c-51d3-46c8-8cac-7117f2c2624e" (UID: "408c162c-51d3-46c8-8cac-7117f2c2624e"). InnerVolumeSpecName "kube-api-access-7khfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:04:52.827524 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.827493    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/408c162c-51d3-46c8-8cac-7117f2c2624e-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:04:52.827524 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:52.827522    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khfw\" (UniqueName: \"kubernetes.io/projected/408c162c-51d3-46c8-8cac-7117f2c2624e-kube-api-access-7khfw\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:04:53.672875 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:53.672843    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"
Jul 29 22:04:53.697954 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:53.697922    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:04:53.707621 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:53.707593    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c8e5f0-5dd66fbb7c-jnn44"]
Jul 29 22:04:54.055395 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:54.055312    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="408c162c-51d3-46c8-8cac-7117f2c2624e" path="/var/lib/kubelet/pods/408c162c-51d3-46c8-8cac-7117f2c2624e/volumes"
Jul 29 22:04:59.358354 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:59.358315    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:04:59.358750 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:04:59.358512    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy" containerID="cri-o://a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18" gracePeriod=30
Jul 29 22:05:04.620281 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.620256    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:05:04.731053 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.731003    2565 generic.go:356] "Generic (PLEG): container finished" podID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerID="a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18" exitCode=0
Jul 29 22:05:04.731226 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.731073    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"
Jul 29 22:05:04.731226 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.731079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerDied","Data":"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"}
Jul 29 22:05:04.731226 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.731186    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj" event={"ID":"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f","Type":"ContainerDied","Data":"5e416e9cf3d71deac63979550e27ffe9a3b47ebb4207a2664e4298f7bc2d005d"}
Jul 29 22:05:04.731226 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.731210    2565 scope.go:117] "RemoveContainer" containerID="a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"
Jul 29 22:05:04.740368 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.740349    2565 scope.go:117] "RemoveContainer" containerID="41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"
Jul 29 22:05:04.742542 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742601    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742621    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742875 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742708    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gndc4\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742875 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.742875 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742809    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:04.742875 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.743092 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742885    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:04.743092 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.742912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo\") pod \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\" (UID: \"65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f\") "
Jul 29 22:05:04.743309 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:04.743391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743344    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.743391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743364    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.743391 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743379    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.743569 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:05:04.743627 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.743566    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data" (OuterVolumeSpecName: "istio-data") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:04.744979 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.744950    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:05:04.745186 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.745161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token" (OuterVolumeSpecName: "istio-token") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:05:04.745287 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.745244    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4" (OuterVolumeSpecName: "kube-api-access-gndc4") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "kube-api-access-gndc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:05:04.745458 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.745427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" (UID: "65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:04.749295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.749273    2565 scope.go:117] "RemoveContainer" containerID="a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"
Jul 29 22:05:04.749587 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:04.749569    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18\": container with ID starting with a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18 not found: ID does not exist" containerID="a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"
Jul 29 22:05:04.749648 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.749598    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18"} err="failed to get container status \"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18\": rpc error: code = NotFound desc = could not find container \"a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18\": container with ID starting with a7b8fa49d416456ef727073152ecfadb234d5ffec20a0762118fbc36f37fcc18 not found: ID does not exist"
Jul 29 22:05:04.749648 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.749617    2565 scope.go:117] "RemoveContainer" containerID="41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"
Jul 29 22:05:04.749871 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:04.749852    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48\": container with ID starting with 41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48 not found: ID does not exist" containerID="41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"
Jul 29 22:05:04.749916 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.749876    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48"} err="failed to get container status \"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48\": rpc error: code = NotFound desc = could not find container \"41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48\": container with ID starting with 41122a511b0d1118cabf54eb2fda2ad8cca9e762656f26f7a39b1dfde8918e48 not found: ID does not exist"
Jul 29 22:05:04.844363 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844329    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.844363 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844357    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.844363 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gndc4\" (UniqueName: \"kubernetes.io/projected/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-kube-api-access-gndc4\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.844596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844375    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.844596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844384    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:04.844596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:04.844392    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:05.055520 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:05.055484    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:05:05.061214 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:05.061179    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c8e5f0-openshift-default-7497f98466-kpckj"]
Jul 29 22:05:06.055749 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:06.055710    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" path="/var/lib/kubelet/pods/65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f/volumes"
Jul 29 22:05:15.422463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.422419    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:05:15.422935 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.422694    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" podUID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" containerName="maas-api" containerID="cri-o://bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879" gracePeriod=30
Jul 29 22:05:15.706957 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.706922    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:05:15.793197 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.793140    2565 generic.go:356] "Generic (PLEG): container finished" podID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" containerID="bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879" exitCode=0
Jul 29 22:05:15.793416 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.793252    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" event={"ID":"a4256062-b3fa-4d01-8f09-e42ec3501b2e","Type":"ContainerDied","Data":"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"}
Jul 29 22:05:15.793416 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.793280    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"
Jul 29 22:05:15.793416 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.793338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2" event={"ID":"a4256062-b3fa-4d01-8f09-e42ec3501b2e","Type":"ContainerDied","Data":"40c505c515c8238c0f5075f190e9c498e6e76d415c121a214238421d2200ed92"}
Jul 29 22:05:15.793416 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.793394    2565 scope.go:117] "RemoveContainer" containerID="bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"
Jul 29 22:05:15.802715 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.802694    2565 scope.go:117] "RemoveContainer" containerID="bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"
Jul 29 22:05:15.803021 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:15.802997    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879\": container with ID starting with bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879 not found: ID does not exist" containerID="bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"
Jul 29 22:05:15.803112 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.803034    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879"} err="failed to get container status \"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879\": rpc error: code = NotFound desc = could not find container \"bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879\": container with ID starting with bdb1769981fc52349616ab4a8c01d8110c38071e5eb43693dfe7c5cd1a3d0879 not found: ID does not exist"
Jul 29 22:05:15.852557 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.852518    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fh6cs\" (UniqueName: \"kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs\") pod \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") "
Jul 29 22:05:15.852774 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.852662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca\") pod \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") "
Jul 29 22:05:15.852774 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.852728    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") pod \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\" (UID: \"a4256062-b3fa-4d01-8f09-e42ec3501b2e\") "
Jul 29 22:05:15.853158 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.853122    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4256062-b3fa-4d01-8f09-e42ec3501b2e" (UID: "a4256062-b3fa-4d01-8f09-e42ec3501b2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:05:15.854835 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.854799    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4256062-b3fa-4d01-8f09-e42ec3501b2e" (UID: "a4256062-b3fa-4d01-8f09-e42ec3501b2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:05:15.854942 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.854892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs" (OuterVolumeSpecName: "kube-api-access-fh6cs") pod "a4256062-b3fa-4d01-8f09-e42ec3501b2e" (UID: "a4256062-b3fa-4d01-8f09-e42ec3501b2e"). InnerVolumeSpecName "kube-api-access-fh6cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:05:15.954367 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.954275    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4256062-b3fa-4d01-8f09-e42ec3501b2e-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:15.954367 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.954312    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6cs\" (UniqueName: \"kubernetes.io/projected/a4256062-b3fa-4d01-8f09-e42ec3501b2e-kube-api-access-fh6cs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:15.954367 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:15.954322    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4256062-b3fa-4d01-8f09-e42ec3501b2e-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:16.120676 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:16.120632    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:05:16.120956 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:16.120918    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-6ab119-678b8c7f47-542j2"]
Jul 29 22:05:18.056487 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:18.056450    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" path="/var/lib/kubelet/pods/a4256062-b3fa-4d01-8f09-e42ec3501b2e/volumes"
Jul 29 22:05:23.002988 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:23.002944    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:05:23.003541 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:23.003201    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy" containerID="cri-o://d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37" gracePeriod=30
Jul 29 22:05:24.001785 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.001746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:05:24.002183 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002169    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" containerName="maas-api"
Jul 29 22:05:24.002234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002184    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" containerName="maas-api"
Jul 29 22:05:24.002234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002206    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.002234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002211    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.002234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002220    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="408c162c-51d3-46c8-8cac-7117f2c2624e" containerName="maas-api"
Jul 29 22:05:24.002234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002225    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="408c162c-51d3-46c8-8cac-7117f2c2624e" containerName="maas-api"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002244    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002249    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002305    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4256062-b3fa-4d01-8f09-e42ec3501b2e" containerName="maas-api"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002312    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="408c162c-51d3-46c8-8cac-7117f2c2624e" containerName="maas-api"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002322    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.002454 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.002331    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="65c8e04b-e7ff-4f86-a6e6-0b53ed4f887f" containerName="istio-proxy"
Jul 29 22:05:24.006625 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.006604    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.009136 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.009112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-c9eccd-openshift-default-dockercfg-4797n\""
Jul 29 22:05:24.034321 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.034281    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:05:24.133451 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133451 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133670 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133568    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133670 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133600    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133670 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgnw5\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133831 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133831 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133924 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.133924 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.133852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235053 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.234996    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235233 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235233 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235233 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235233 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgnw5\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235474 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235474 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235474 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235361    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235474 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.235999 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.235968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.236157 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.236075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.236157 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.236095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.236299 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.236183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.237497 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.237475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.237595 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.237572    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.248000 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.247959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgnw5\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.248146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.247976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token\") pod \"e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") " pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.317139 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.317027    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:24.469995 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.469963    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:05:24.471790 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:05:24.471766    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode04582e1_a195_4899_b1d3_6ef1f7538bc9.slice/crio-e4576e58c73ac7cd02e20736938d9453bb8ffe73de20151311b7497237d182be WatchSource:0}: Error finding container e4576e58c73ac7cd02e20736938d9453bb8ffe73de20151311b7497237d182be: Status 404 returned error can't find the container with id e4576e58c73ac7cd02e20736938d9453bb8ffe73de20151311b7497237d182be
Jul 29 22:05:24.473966 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.473931    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:24.474064 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.473997    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:24.474064 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.474027    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:24.842496 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.842457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerStarted","Data":"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"}
Jul 29 22:05:24.842707 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.842512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerStarted","Data":"e4576e58c73ac7cd02e20736938d9453bb8ffe73de20151311b7497237d182be"}
Jul 29 22:05:24.865908 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:24.865858    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podStartSLOduration=1.8658437289999998 podStartE2EDuration="1.865843729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:24.863032872 +0000 UTC m=+2941.334612007" watchObservedRunningTime="2026-07-29 22:05:24.865843729 +0000 UTC m=+2941.337422805"
Jul 29 22:05:25.318000 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:25.317964    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:25.323055 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:25.323014    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:25.848598 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:25.848566    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:25.849564 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:25.849542    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:28.101931 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.101895    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:05:28.105680 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.105651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.108532 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.108505    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-c9eccd-serving-cert\""
Jul 29 22:05:28.125168 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.125132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:05:28.275645 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.275581    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:05:28.276676 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.276656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.276730 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.276713    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.276839 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.276819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2x88\" (UniqueName: \"kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.377993 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.377956    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.377993 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378111    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5tvh\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378261 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378584 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378305    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:28.378584 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378355    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert\") pod \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\" (UID: \"c3e4c86b-a16c-49f6-b845-fa7f50807bda\") "
Jul 29 22:05:28.378584 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:28.378743 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2x88\" (UniqueName: \"kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.378743 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:28.378859 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378764    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.378916 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.378916 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378884    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:05:28.379025 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:28.378990    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-serving-cert: secret "maas-api-e2e-shared-a-c9eccd-serving-cert" not found
Jul 29 22:05:28.379025 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.378997    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.379025 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.379016    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.379211 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.379031    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.379211 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.379063    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.379211 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:28.379086    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls podName:63a8a3cb-d1b8-4575-b42a-8e33717ff6a0 nodeName:}" failed. No retries permitted until 2026-07-29 22:05:28.879064728 +0000 UTC m=+2945.350643787 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls") pod "maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" (UID: "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0") : secret "maas-api-e2e-shared-a-c9eccd-serving-cert" not found
Jul 29 22:05:28.379211 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.379107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:28.379583 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.379561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.380459 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.380439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:05:28.380568 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.380543    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:05:28.380764 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.380748    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:05:28.380898 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.380873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh" (OuterVolumeSpecName: "kube-api-access-d5tvh") pod "c3e4c86b-a16c-49f6-b845-fa7f50807bda" (UID: "c3e4c86b-a16c-49f6-b845-fa7f50807bda"). InnerVolumeSpecName "kube-api-access-d5tvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:05:28.391346 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.391320    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2x88\" (UniqueName: \"kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.480230 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.480195    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.480230 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.480228    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.480230 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.480240    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.480492 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.480254    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5tvh\" (UniqueName: \"kubernetes.io/projected/c3e4c86b-a16c-49f6-b845-fa7f50807bda-kube-api-access-d5tvh\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.480492 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.480276    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3e4c86b-a16c-49f6-b845-fa7f50807bda-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:05:28.870361 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.870324    2565 generic.go:356] "Generic (PLEG): container finished" podID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerID="d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37" exitCode=0
Jul 29 22:05:28.870550 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.870374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"
Jul 29 22:05:28.870550 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.870413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerDied","Data":"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"}
Jul 29 22:05:28.870550 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.870468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x" event={"ID":"c3e4c86b-a16c-49f6-b845-fa7f50807bda","Type":"ContainerDied","Data":"482c33fda113dd251d29b5762d806b4eb2fb70f6748478b31e970e8219b311aa"}
Jul 29 22:05:28.870550 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.870491    2565 scope.go:117] "RemoveContainer" containerID="d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"
Jul 29 22:05:28.880075 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.880036    2565 scope.go:117] "RemoveContainer" containerID="c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"
Jul 29 22:05:28.885532 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.885509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.888140 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.888115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") pod \"maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:28.888800 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.888774    2565 scope.go:117] "RemoveContainer" containerID="d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"
Jul 29 22:05:28.889093 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:28.889075    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37\": container with ID starting with d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37 not found: ID does not exist" containerID="d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"
Jul 29 22:05:28.889146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.889101    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37"} err="failed to get container status \"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37\": rpc error: code = NotFound desc = could not find container \"d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37\": container with ID starting with d3650c20c5e5f95fe466ef26609f3244da3b9e869a6205ba7776c0b42d01fa37 not found: ID does not exist"
Jul 29 22:05:28.889146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.889116    2565 scope.go:117] "RemoveContainer" containerID="c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"
Jul 29 22:05:28.889386 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:28.889357    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30\": container with ID starting with c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30 not found: ID does not exist" containerID="c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"
Jul 29 22:05:28.889445 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.889392    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30"} err="failed to get container status \"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30\": rpc error: code = NotFound desc = could not find container \"c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30\": container with ID starting with c1e2eafa01e901216316567a266e8f14423c74df103cd9995e080cc601eabe30 not found: ID does not exist"
Jul 29 22:05:28.905087 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.905036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:05:28.907209 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:28.907187    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ab119-openshift-default-5c4988f9b6-9wd7x"]
Jul 29 22:05:29.017302 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:29.017272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:29.396215 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:29.396188    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:05:29.398542 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:05:29.398505    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63a8a3cb_d1b8_4575_b42a_8e33717ff6a0.slice/crio-e24a2b0c0bb92415f53bb1b7c4148556ecc5fd29ebae10e72e934706b61af29d WatchSource:0}: Error finding container e24a2b0c0bb92415f53bb1b7c4148556ecc5fd29ebae10e72e934706b61af29d: Status 404 returned error can't find the container with id e24a2b0c0bb92415f53bb1b7c4148556ecc5fd29ebae10e72e934706b61af29d
Jul 29 22:05:29.877731 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:29.877691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" event={"ID":"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0","Type":"ContainerStarted","Data":"e24a2b0c0bb92415f53bb1b7c4148556ecc5fd29ebae10e72e934706b61af29d"}
Jul 29 22:05:30.055514 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:30.055474    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" path="/var/lib/kubelet/pods/c3e4c86b-a16c-49f6-b845-fa7f50807bda/volumes"
Jul 29 22:05:30.884463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:30.884427    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" event={"ID":"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0","Type":"ContainerStarted","Data":"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"}
Jul 29 22:05:30.884870 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:30.884544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:30.907244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:30.907192    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" podStartSLOduration=2.40597733 podStartE2EDuration="2.907178651s" totalImagesPullingTime="501.201321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:30.905820147 +0000 UTC m=+2947.377399226" watchObservedRunningTime="2026-07-29 22:05:30.907178651 +0000 UTC m=+2947.378757727"
Jul 29 22:05:36.897187 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:36.897148    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:05:39.933989 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.933953    2565 generic.go:356] "Generic (PLEG): container finished" podID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerID="59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8" exitCode=0
Jul 29 22:05:39.934540 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.934027    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerDied","Data":"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"}
Jul 29 22:05:39.934614 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.934592    2565 scope.go:117] "RemoveContainer" containerID="59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"
Jul 29 22:05:39.935659 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.935631    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:39.935773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.935696    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:39.935773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:39.935731    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:40.849582 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:40.849474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:40.942107 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:40.941328    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerStarted","Data":"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"}
Jul 29 22:05:41.317400 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:41.317367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:41.318739 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:41.318707    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 29 22:05:41.318864 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:41.318762    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 29 22:05:42.317784 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:42.317748    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 29 22:05:42.318181 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:42.317807    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 29 22:05:43.317919 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:43.317880    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 29 22:05:43.318340 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:43.317949    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 29 22:05:44.318453 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:44.318412    2565 patch_prober.go:28] interesting pod/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 29 22:05:44.318838 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:44.318499    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 29 22:05:45.321788 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:45.321751    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:45.322238 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:45.322108    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:45.322865 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:45.322846    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:05:52.688810 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.688726    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:05:52.689316 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.689274    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:52.689316 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.689290    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:52.689415 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.689404    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:52.692735 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.692718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.698928 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.698907    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-ca895e-openshift-default-dockercfg-pn5c8\""
Jul 29 22:05:52.712552 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.712516    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:05:52.810059 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.809926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810059 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.809979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdx7\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810261    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810527 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.810527 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.810401    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911395 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911395 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911511    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdx7\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911741    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911801    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.911874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.912100 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.911880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.912144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.912123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.912398 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.912379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.913862 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.913844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.913937 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.913921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.921222 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.921191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:52.922251 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:52.922230    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdx7\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7\") pod \"e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") " pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:53.003660 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:53.003577    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:53.156419 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:53.156391    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:05:53.158953 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:05:53.158922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb49e65_594a_402f_a4df_4ab35a491048.slice/crio-2dedbea15c2fbd52ea3ed8fa837935c642ceb29021194c7803560c30a011c9af WatchSource:0}: Error finding container 2dedbea15c2fbd52ea3ed8fa837935c642ceb29021194c7803560c30a011c9af: Status 404 returned error can't find the container with id 2dedbea15c2fbd52ea3ed8fa837935c642ceb29021194c7803560c30a011c9af
Jul 29 22:05:53.161556 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:53.161505    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:53.161661 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:53.161584    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:53.161661 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:53.161612    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:05:54.021714 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:54.021669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerStarted","Data":"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"}
Jul 29 22:05:54.022182 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:54.021739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerStarted","Data":"2dedbea15c2fbd52ea3ed8fa837935c642ceb29021194c7803560c30a011c9af"}
Jul 29 22:05:54.065990 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:54.065923    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podStartSLOduration=2.065902759 podStartE2EDuration="2.065902759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:54.063634448 +0000 UTC m=+2970.535213525" watchObservedRunningTime="2026-07-29 22:05:54.065902759 +0000 UTC m=+2970.537481836"
Jul 29 22:05:55.004395 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:55.004356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:55.009422 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:55.009395    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:55.027389 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:55.027360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:55.028379 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:55.028360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:05:57.331839 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.331792    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:05:57.332346 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.332251    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:57.332346 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.332264    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:57.332346 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.332340    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3e4c86b-a16c-49f6-b845-fa7f50807bda" containerName="istio-proxy"
Jul 29 22:05:57.335813 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.335788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.338168 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.338146    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ca895e-serving-cert\""
Jul 29 22:05:57.346817 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.346792    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:05:57.461029 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.460987    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bsw6\" (UniqueName: \"kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.461246 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.461176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.461246 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.461238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.562204 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.562168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.562392 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.562230    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.562392 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.562278    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bsw6\" (UniqueName: \"kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.562475 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:57.562393    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-serving-cert: secret "maas-api-e2e-shared-b-ca895e-serving-cert" not found
Jul 29 22:05:57.562510 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:05:57.562488    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls podName:7dab7830-950a-439a-a7b1-cea70c9b29fc nodeName:}" failed. No retries permitted until 2026-07-29 22:05:58.06246638 +0000 UTC m=+2974.534045434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls") pod "maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" (UID: "7dab7830-950a-439a-a7b1-cea70c9b29fc") : secret "maas-api-e2e-shared-b-ca895e-serving-cert" not found
Jul 29 22:05:57.562917 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.562898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:57.571579 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:57.571552    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bsw6\" (UniqueName: \"kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:58.068220 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:58.068189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:58.070809 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:58.070774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") pod \"maas-api-e2e-shared-b-ca895e-74845c5757-bgs22\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:58.248906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:58.248830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:58.427198 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:58.427169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:05:58.429534 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:05:58.429506    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dab7830_950a_439a_a7b1_cea70c9b29fc.slice/crio-e18a9538a0f3d1d7672db7aa262c90da521954493140a455ac09541751fd7e57 WatchSource:0}: Error finding container e18a9538a0f3d1d7672db7aa262c90da521954493140a455ac09541751fd7e57: Status 404 returned error can't find the container with id e18a9538a0f3d1d7672db7aa262c90da521954493140a455ac09541751fd7e57
Jul 29 22:05:59.054256 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:59.054219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" event={"ID":"7dab7830-950a-439a-a7b1-cea70c9b29fc","Type":"ContainerStarted","Data":"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"}
Jul 29 22:05:59.054427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:59.054307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" event={"ID":"7dab7830-950a-439a-a7b1-cea70c9b29fc","Type":"ContainerStarted","Data":"e18a9538a0f3d1d7672db7aa262c90da521954493140a455ac09541751fd7e57"}
Jul 29 22:05:59.054427 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:59.054349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:05:59.076510 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:05:59.076457    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" podStartSLOduration=1.524941862 podStartE2EDuration="2.076442273s" totalImagesPullingTime="551.500411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:59.074279518 +0000 UTC m=+2975.545858594" watchObservedRunningTime="2026-07-29 22:05:59.076442273 +0000 UTC m=+2975.548021349"
Jul 29 22:06:05.068287 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:05.068249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:06:11.068223 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:11.068176    2565 patch_prober.go:28] interesting pod/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 22:06:11.068680 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:11.068259    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 22:06:12.123955 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.123924    2565 generic.go:356] "Generic (PLEG): container finished" podID="deb49e65-594a-402f-a4df-4ab35a491048" containerID="ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c" exitCode=0
Jul 29 22:06:12.124373 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.123963    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerDied","Data":"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"}
Jul 29 22:06:12.124446 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.124430    2565 scope.go:117] "RemoveContainer" containerID="ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"
Jul 29 22:06:12.125538 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.125505    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:06:12.125599 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.125579    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:06:12.125636 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:12.125611    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:06:13.131943 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:13.131906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerStarted","Data":"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"}
Jul 29 22:06:14.004772 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:14.004733    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:06:14.005033 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:14.005011    2565 patch_prober.go:28] interesting pod/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 22:06:14.005117 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:14.005089    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 22:06:15.004425 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:15.004382    2565 patch_prober.go:28] interesting pod/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 22:06:15.004922 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:15.004455    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 22:06:16.005037 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:16.005000    2565 patch_prober.go:28] interesting pod/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 22:06:16.005461 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:16.005087    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 22:06:17.004872 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:17.004833    2565 patch_prober.go:28] interesting pod/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 29 22:06:17.005075 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:17.004893    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 29 22:06:18.007870 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:18.007840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:06:18.008278 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:18.008199    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:06:18.008905 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:06:18.008885    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:10:24.471589 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:10:24.471516    2565 scope.go:117] "RemoveContainer" containerID="bc0f1e88f86e12100d0c30d61f8facb9c465616229cecbfbca307a12baa72a20"
Jul 29 22:15:03.190548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:15:03.190467    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"]
Jul 29 22:15:03.193602 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:15:03.193569    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29756010-qs286"]
Jul 29 22:15:04.055015 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:15:04.054968    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25f9bfda-4143-4878-85d2-b455b5402371" path="/var/lib/kubelet/pods/25f9bfda-4143-4878-85d2-b455b5402371/volumes"
Jul 29 22:15:24.525986 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:15:24.525947    2565 scope.go:117] "RemoveContainer" containerID="c9d4f248dbddd7fbd9c0edf094b8aaf8a748dcdca450e784bc130eaa4a8d862a"
Jul 29 22:17:20.632677 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.632638    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:17:20.636162 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.636140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.638353 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.638322    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ee9d15-serving-cert\""
Jul 29 22:17:20.646925 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.646899    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:17:20.721138 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.721104    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.721325 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.721190    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.721325 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.721210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccjjd\" (UniqueName: \"kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.822008 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.821969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.822008 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.822006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccjjd\" (UniqueName: \"kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.822279 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.822110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.822279 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:17:20.822218    2565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-serving-cert: secret "maas-api-e2e-ipp-a-ee9d15-serving-cert" not found
Jul 29 22:17:20.822279 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:17:20.822275    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls podName:0e698bfb-2567-4f6b-ac61-4f5483335f8f nodeName:}" failed. No retries permitted until 2026-07-29 22:17:21.322259284 +0000 UTC m=+3657.793838338 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls") pod "maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" (UID: "0e698bfb-2567-4f6b-ac61-4f5483335f8f") : secret "maas-api-e2e-ipp-a-ee9d15-serving-cert" not found
Jul 29 22:17:20.822678 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.822658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:20.832947 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:20.832918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccjjd\" (UniqueName: \"kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:21.326415 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.326377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:21.328920 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.328892    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:21.549594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.549560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:21.705203 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.705169    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:17:21.708619 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:17:21.708590    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e698bfb_2567_4f6b_ac61_4f5483335f8f.slice/crio-e74c04ea4549f2e341ec32b6c67fc3f02e760d0448f4884385258c5f6ad640fd WatchSource:0}: Error finding container e74c04ea4549f2e341ec32b6c67fc3f02e760d0448f4884385258c5f6ad640fd: Status 404 returned error can't find the container with id e74c04ea4549f2e341ec32b6c67fc3f02e760d0448f4884385258c5f6ad640fd
Jul 29 22:17:21.710057 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.710027    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 29 22:17:21.724187 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:21.724158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" event={"ID":"0e698bfb-2567-4f6b-ac61-4f5483335f8f","Type":"ContainerStarted","Data":"e74c04ea4549f2e341ec32b6c67fc3f02e760d0448f4884385258c5f6ad640fd"}
Jul 29 22:17:25.746514 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:25.746480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" event={"ID":"0e698bfb-2567-4f6b-ac61-4f5483335f8f","Type":"ContainerStarted","Data":"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"}
Jul 29 22:17:25.746904 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:25.746558    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:25.766070 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:25.765992    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" podStartSLOduration=2.48922703 podStartE2EDuration="5.765974622s" totalImagesPullingTime="3.276747592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:25.763901609 +0000 UTC m=+3662.235480690" watchObservedRunningTime="2026-07-29 22:17:25.765974622 +0000 UTC m=+3662.237553699"
Jul 29 22:17:31.756412 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:31.756382    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:17:45.603490 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.603454    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:17:45.607581 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.607556    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.609963 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.609936    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-b26bbb-openshift-default-dockercfg-pbbhx\""
Jul 29 22:17:45.622171 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.622142    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:17:45.776947 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.776909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79v5h\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.776947 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.776963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777361 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777361 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777361 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.777506 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.777374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.878952 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.878860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.878950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.878986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879070    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879144 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79v5h\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879354    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879463 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879654 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.879929 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.879912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.881335 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.881311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.881659 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.881546    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.887999 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.887974    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79v5h\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.888127 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.888073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token\") pod \"e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") " pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:45.919825 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:45.919790    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:46.061230 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.061206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:17:46.063171 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:17:46.063142    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46867301_08d7_4fd3_ab7b_87672734e344.slice/crio-542a8d158e49196824079bb25a4298ac53158b63ee589f5446d87e124f22fda6 WatchSource:0}: Error finding container 542a8d158e49196824079bb25a4298ac53158b63ee589f5446d87e124f22fda6: Status 404 returned error can't find the container with id 542a8d158e49196824079bb25a4298ac53158b63ee589f5446d87e124f22fda6
Jul 29 22:17:46.065502 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.065454    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:17:46.065579 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.065539    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:17:46.065579 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.065565    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:17:46.862787 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.862707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerStarted","Data":"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"}
Jul 29 22:17:46.862787 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.862758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerStarted","Data":"542a8d158e49196824079bb25a4298ac53158b63ee589f5446d87e124f22fda6"}
Jul 29 22:17:46.890073 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.886285    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podStartSLOduration=1.886263686 podStartE2EDuration="1.886263686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:46.883035612 +0000 UTC m=+3683.354614688" watchObservedRunningTime="2026-07-29 22:17:46.886263686 +0000 UTC m=+3683.357842762"
Jul 29 22:17:46.920538 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:46.920507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:47.925419 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:47.925386    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:48.872004 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:48.871965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:48.873067 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:48.873029    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:17:49.908232 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:49.908198    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:17:49.912908 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:49.912882    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:49.916500 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:49.915755    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b26bbb-serving-cert\""
Jul 29 22:17:49.924379 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:49.924352    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:17:50.028295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.028253    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbv8x\" (UniqueName: \"kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.028295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.028299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.028562 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.028452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.130117 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.130081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbv8x\" (UniqueName: \"kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.130292 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.130123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.130292 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.130228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.130983 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.130946    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.132615 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.132585    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.138316 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.138293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbv8x\" (UniqueName: \"kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x\") pod \"maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.227464 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.227427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.369797 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.369763    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:17:50.370730 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:17:50.370696    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda127c57c_1e85_4399_bb01_a5125f07b9d9.slice/crio-8f0cf7bc518ee3884ad2e41e7710a4638960a3b6d731bde16cca7f3b32125085 WatchSource:0}: Error finding container 8f0cf7bc518ee3884ad2e41e7710a4638960a3b6d731bde16cca7f3b32125085: Status 404 returned error can't find the container with id 8f0cf7bc518ee3884ad2e41e7710a4638960a3b6d731bde16cca7f3b32125085
Jul 29 22:17:50.883793 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.883754    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" event={"ID":"a127c57c-1e85-4399-bb01-a5125f07b9d9","Type":"ContainerStarted","Data":"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"}
Jul 29 22:17:50.883982 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.883808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" event={"ID":"a127c57c-1e85-4399-bb01-a5125f07b9d9","Type":"ContainerStarted","Data":"8f0cf7bc518ee3884ad2e41e7710a4638960a3b6d731bde16cca7f3b32125085"}
Jul 29 22:17:50.883982 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.883835    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:17:50.901193 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:50.901148    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" podStartSLOduration=1.544352492 podStartE2EDuration="1.901132432s" totalImagesPullingTime="356.77994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:50.900358131 +0000 UTC m=+3687.371937205" watchObservedRunningTime="2026-07-29 22:17:50.901132432 +0000 UTC m=+3687.372711506"
Jul 29 22:17:56.894698 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:17:56.894665    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:18:04.914195 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:04.914148    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 29 22:18:04.914569 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:04.914219    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 29 22:18:06.974596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.974561    2565 generic.go:356] "Generic (PLEG): container finished" podID="46867301-08d7-4fd3-ab7b-87672734e344" containerID="2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b" exitCode=0
Jul 29 22:18:06.975029 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.974639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerDied","Data":"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"}
Jul 29 22:18:06.975101 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.975083    2565 scope.go:117] "RemoveContainer" containerID="2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"
Jul 29 22:18:06.976129 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.976101    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:18:06.976234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.976183    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:18:06.976295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:06.976232    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:18:07.982300 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:07.982262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerStarted","Data":"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"}
Jul 29 22:18:08.920601 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:08.920562    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:18:08.920814 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:08.920791    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 22:18:08.920874 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:08.920854    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 22:18:09.921016 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:09.920980    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 22:18:09.921429 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:09.921035    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 22:18:10.920762 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:10.920723    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Jul 29 22:18:10.920948 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:10.920782    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Jul 29 22:18:11.924498 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:11.924468    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:18:11.924952 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:11.924892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:18:11.925724 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:11.925705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:18:20.922851 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:18:20.922823    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:19:16.348715 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:16.348635    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:19:17.140178 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.140138    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:19:17.144033 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.144008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.146448 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.146424    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-eafc3c-openshift-default-dockercfg-4gbpg\""
Jul 29 22:19:17.164256 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.164223    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:19:17.260501 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.260782 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.260782 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.260782 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.261016 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.261016 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.261016 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhpzh\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.261016 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.260890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.261281 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.261016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.361863 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.361819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.361880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.361910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhpzh\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.361955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362141    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362360 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362941 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362371    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362941 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.362941 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.362658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.363119 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.363013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.364426 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.364405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.364644 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.364627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.372383 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.372361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhpzh\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.372599 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.372579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token\") pod \"e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") " pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.455494 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.455450    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:17.616826 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.616794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:19:17.619085 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:19:17.619032    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9804e4cd_8245_4331_94e2_c6e48611c429.slice/crio-f33b0c0d26ed1e223dea137da16513c9472de9f9b9a99bd7fa1f55d125be1e40 WatchSource:0}: Error finding container f33b0c0d26ed1e223dea137da16513c9472de9f9b9a99bd7fa1f55d125be1e40: Status 404 returned error can't find the container with id f33b0c0d26ed1e223dea137da16513c9472de9f9b9a99bd7fa1f55d125be1e40
Jul 29 22:19:17.621804 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.621770    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:17.621912 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.621836    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:17.621912 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:17.621868    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:18.363298 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:18.363263    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerStarted","Data":"6071893cd9da4c57ec573cbe565a454ec4a693d0a762aeac070819aba705b1fc"}
Jul 29 22:19:18.363796 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:18.363329    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerStarted","Data":"f33b0c0d26ed1e223dea137da16513c9472de9f9b9a99bd7fa1f55d125be1e40"}
Jul 29 22:19:18.385632 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:18.385571    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podStartSLOduration=1.38555193 podStartE2EDuration="1.38555193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:19:18.38330058 +0000 UTC m=+3774.854879682" watchObservedRunningTime="2026-07-29 22:19:18.38555193 +0000 UTC m=+3774.857131068"
Jul 29 22:19:18.455788 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:18.455744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:18.461089 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:18.461037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:19.369208 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:19.369178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:19.370304 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:19.370285    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:34.370979 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.370934    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 29 22:19:34.371383 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.371002    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 29 22:19:34.457638 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.457603    2565 generic.go:356] "Generic (PLEG): container finished" podID="9804e4cd-8245-4331-94e2-c6e48611c429" containerID="6071893cd9da4c57ec573cbe565a454ec4a693d0a762aeac070819aba705b1fc" exitCode=0
Jul 29 22:19:34.457816 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.457675    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerDied","Data":"6071893cd9da4c57ec573cbe565a454ec4a693d0a762aeac070819aba705b1fc"}
Jul 29 22:19:34.458185 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.458170    2565 scope.go:117] "RemoveContainer" containerID="6071893cd9da4c57ec573cbe565a454ec4a693d0a762aeac070819aba705b1fc"
Jul 29 22:19:34.459287 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.459261    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:34.459353 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.459338    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:34.459396 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:34.459371    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 29 22:19:35.471388 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:35.471351    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerStarted","Data":"fa651c42af38502942bc46dc4ad61be667c0409acf49540ba383bcebc64e1126"}
Jul 29 22:19:36.456295 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:36.456252    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:36.456721 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:36.456688    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 29 22:19:36.456946 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:36.456918    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 29 22:19:37.456340 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:37.456292    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 29 22:19:37.456849 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:37.456364    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 29 22:19:38.455901 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:38.455858    2565 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Jul 29 22:19:38.456136 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:38.455931    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Jul 29 22:19:39.459768 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:39.459737    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:39.460163 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:39.460072    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:39.460650 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:39.460631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:19:58.476585 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:58.476545    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:19:58.477135 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:19:58.476837    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy" containerID="cri-o://fa651c42af38502942bc46dc4ad61be667c0409acf49540ba383bcebc64e1126" gracePeriod=30
Jul 29 22:20:03.636992 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.636948    2565 generic.go:356] "Generic (PLEG): container finished" podID="9804e4cd-8245-4331-94e2-c6e48611c429" containerID="fa651c42af38502942bc46dc4ad61be667c0409acf49540ba383bcebc64e1126" exitCode=0
Jul 29 22:20:03.637404 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.637015    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerDied","Data":"fa651c42af38502942bc46dc4ad61be667c0409acf49540ba383bcebc64e1126"}
Jul 29 22:20:03.637404 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.637123    2565 scope.go:117] "RemoveContainer" containerID="6071893cd9da4c57ec573cbe565a454ec4a693d0a762aeac070819aba705b1fc"
Jul 29 22:20:03.748303 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.748245    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:20:03.868839 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.868801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.868855    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.868883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.868908    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.868932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869084    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869177 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhpzh\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869600 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket\") pod \"9804e4cd-8245-4331-94e2-c6e48611c429\" (UID: \"9804e4cd-8245-4331-94e2-c6e48611c429\") "
Jul 29 22:20:03.869600 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:20:03.869600 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:20:03.869600 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869535    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:20:03.869906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869757    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.869906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869778    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9804e4cd-8245-4331-94e2-c6e48611c429-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.869906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869793    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.869906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869852    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data" (OuterVolumeSpecName: "istio-data") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:20:03.870070 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.869993    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:20:03.871303 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.871273    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:20:03.871398 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.871335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token" (OuterVolumeSpecName: "istio-token") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:20:03.871811 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.871789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh" (OuterVolumeSpecName: "kube-api-access-vhpzh") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "kube-api-access-vhpzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:20:03.871872 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.871835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9804e4cd-8245-4331-94e2-c6e48611c429" (UID: "9804e4cd-8245-4331-94e2-c6e48611c429"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:20:03.970816 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970775    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhpzh\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-kube-api-access-vhpzh\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.970816 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970808    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.970816 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970819    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.971131 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970828    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9804e4cd-8245-4331-94e2-c6e48611c429-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.971131 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970839    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9804e4cd-8245-4331-94e2-c6e48611c429-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:03.971131 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:03.970851    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9804e4cd-8245-4331-94e2-c6e48611c429-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:04.643848 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:04.643818    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"
Jul 29 22:20:04.644362 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:04.643847    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v" event={"ID":"9804e4cd-8245-4331-94e2-c6e48611c429","Type":"ContainerDied","Data":"f33b0c0d26ed1e223dea137da16513c9472de9f9b9a99bd7fa1f55d125be1e40"}
Jul 29 22:20:04.644362 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:04.643903    2565 scope.go:117] "RemoveContainer" containerID="fa651c42af38502942bc46dc4ad61be667c0409acf49540ba383bcebc64e1126"
Jul 29 22:20:04.660671 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:04.660643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:20:04.664722 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:04.664697    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-eafc3c-openshift-default-6957b55c54-7xm8v"]
Jul 29 22:20:06.055298 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:06.055256    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" path="/var/lib/kubelet/pods/9804e4cd-8245-4331-94e2-c6e48611c429/volumes"
Jul 29 22:20:17.887505 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:17.887466    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:20:17.888007 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:17.887777    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" podUID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" containerName="maas-api" containerID="cri-o://feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8" gracePeriod=30
Jul 29 22:20:18.175630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.175604    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:20:18.324386 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.324344    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccjjd\" (UniqueName: \"kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd\") pod \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") "
Jul 29 22:20:18.324793 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.324768    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca\") pod \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") "
Jul 29 22:20:18.324927 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.324834    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") pod \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\" (UID: \"0e698bfb-2567-4f6b-ac61-4f5483335f8f\") "
Jul 29 22:20:18.325418 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.325389    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e698bfb-2567-4f6b-ac61-4f5483335f8f" (UID: "0e698bfb-2567-4f6b-ac61-4f5483335f8f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:20:18.326898 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.326869    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd" (OuterVolumeSpecName: "kube-api-access-ccjjd") pod "0e698bfb-2567-4f6b-ac61-4f5483335f8f" (UID: "0e698bfb-2567-4f6b-ac61-4f5483335f8f"). InnerVolumeSpecName "kube-api-access-ccjjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:20:18.327124 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.327103    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e698bfb-2567-4f6b-ac61-4f5483335f8f" (UID: "0e698bfb-2567-4f6b-ac61-4f5483335f8f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:20:18.426244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.426208    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccjjd\" (UniqueName: \"kubernetes.io/projected/0e698bfb-2567-4f6b-ac61-4f5483335f8f-kube-api-access-ccjjd\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:18.426244 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.426241    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e698bfb-2567-4f6b-ac61-4f5483335f8f-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:18.426443 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.426251    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e698bfb-2567-4f6b-ac61-4f5483335f8f-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:20:18.721741 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.721655    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" containerID="feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8" exitCode=0
Jul 29 22:20:18.721741 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.721696    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"
Jul 29 22:20:18.721920 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.721746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" event={"ID":"0e698bfb-2567-4f6b-ac61-4f5483335f8f","Type":"ContainerDied","Data":"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"}
Jul 29 22:20:18.721920 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.721804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p" event={"ID":"0e698bfb-2567-4f6b-ac61-4f5483335f8f","Type":"ContainerDied","Data":"e74c04ea4549f2e341ec32b6c67fc3f02e760d0448f4884385258c5f6ad640fd"}
Jul 29 22:20:18.721920 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.721827    2565 scope.go:117] "RemoveContainer" containerID="feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"
Jul 29 22:20:18.732318 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.732294    2565 scope.go:117] "RemoveContainer" containerID="feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"
Jul 29 22:20:18.732581 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:20:18.732563    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8\": container with ID starting with feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8 not found: ID does not exist" containerID="feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"
Jul 29 22:20:18.732622 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.732589    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8"} err="failed to get container status \"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8\": rpc error: code = NotFound desc = could not find container \"feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8\": container with ID starting with feec9d9f8f8184327f2adeb572d42299a102a562bc3d71776da5ab0eb59101d8 not found: ID does not exist"
Jul 29 22:20:18.746160 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.746122    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:20:18.749612 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:18.749574    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ee9d15-6d99bbcf6b-vlp6p"]
Jul 29 22:20:20.055220 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:20:20.055190    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" path="/var/lib/kubelet/pods/0e698bfb-2567-4f6b-ac61-4f5483335f8f/volumes"
Jul 29 22:21:23.031653 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.031614    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:21:23.032169 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.031906    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" podUID="a127c57c-1e85-4399-bb01-a5125f07b9d9" containerName="maas-api" containerID="cri-o://3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c" gracePeriod=30
Jul 29 22:21:23.318940 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.318911    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:21:23.403868 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.403836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbv8x\" (UniqueName: \"kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x\") pod \"a127c57c-1e85-4399-bb01-a5125f07b9d9\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") "
Jul 29 22:21:23.404084 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.403887    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls\") pod \"a127c57c-1e85-4399-bb01-a5125f07b9d9\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") "
Jul 29 22:21:23.404084 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.403946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca\") pod \"a127c57c-1e85-4399-bb01-a5125f07b9d9\" (UID: \"a127c57c-1e85-4399-bb01-a5125f07b9d9\") "
Jul 29 22:21:23.404400 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.404370    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a127c57c-1e85-4399-bb01-a5125f07b9d9" (UID: "a127c57c-1e85-4399-bb01-a5125f07b9d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:21:23.406090 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.406058    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x" (OuterVolumeSpecName: "kube-api-access-rbv8x") pod "a127c57c-1e85-4399-bb01-a5125f07b9d9" (UID: "a127c57c-1e85-4399-bb01-a5125f07b9d9"). InnerVolumeSpecName "kube-api-access-rbv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:23.406539 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.406511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a127c57c-1e85-4399-bb01-a5125f07b9d9" (UID: "a127c57c-1e85-4399-bb01-a5125f07b9d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:21:23.504906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.504867    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbv8x\" (UniqueName: \"kubernetes.io/projected/a127c57c-1e85-4399-bb01-a5125f07b9d9-kube-api-access-rbv8x\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:23.504906 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.504902    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a127c57c-1e85-4399-bb01-a5125f07b9d9-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:23.505155 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:23.504914    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a127c57c-1e85-4399-bb01-a5125f07b9d9-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:24.074992 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.074964    2565 generic.go:356] "Generic (PLEG): container finished" podID="a127c57c-1e85-4399-bb01-a5125f07b9d9" containerID="3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c" exitCode=0
Jul 29 22:21:24.075364 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.075006    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"
Jul 29 22:21:24.075364 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.075063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" event={"ID":"a127c57c-1e85-4399-bb01-a5125f07b9d9","Type":"ContainerDied","Data":"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"}
Jul 29 22:21:24.075364 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.075124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5" event={"ID":"a127c57c-1e85-4399-bb01-a5125f07b9d9","Type":"ContainerDied","Data":"8f0cf7bc518ee3884ad2e41e7710a4638960a3b6d731bde16cca7f3b32125085"}
Jul 29 22:21:24.075364 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.075137    2565 scope.go:117] "RemoveContainer" containerID="3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"
Jul 29 22:21:24.083907 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.083886    2565 scope.go:117] "RemoveContainer" containerID="3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"
Jul 29 22:21:24.084211 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:24.084190    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c\": container with ID starting with 3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c not found: ID does not exist" containerID="3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"
Jul 29 22:21:24.084285 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.084220    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c"} err="failed to get container status \"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c\": rpc error: code = NotFound desc = could not find container \"3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c\": container with ID starting with 3e5fed6df2627ba1ddc263884c947d71caf5be0d9778fb2d710c6ee6855add0c not found: ID does not exist"
Jul 29 22:21:24.091342 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.091313    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:21:24.096128 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:24.096105    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b26bbb-7b9c4cbc7b-4d2v5"]
Jul 29 22:21:26.055365 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:26.055332    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a127c57c-1e85-4399-bb01-a5125f07b9d9" path="/var/lib/kubelet/pods/a127c57c-1e85-4399-bb01-a5125f07b9d9/volumes"
Jul 29 22:21:29.550930 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:29.550894    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:21:29.551476 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:29.551161    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy" containerID="cri-o://4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8" gracePeriod=30
Jul 29 22:21:34.808450 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.808422    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:21:34.928635 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.928827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928696    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.928827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.928827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.928827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928795    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.928827 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928829    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79v5h\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.929146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.929146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928898    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.929146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:34.929146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.928934    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy\") pod \"46867301-08d7-4fd3-ab7b-87672734e344\" (UID: \"46867301-08d7-4fd3-ab7b-87672734e344\") "
Jul 29 22:21:34.929146 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929012    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:34.929410 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929219    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:21:34.929548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929501    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46867301-08d7-4fd3-ab7b-87672734e344-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:34.929548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929537    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:34.929548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:34.929548 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.929553    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:34.930319 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.930289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data" (OuterVolumeSpecName: "istio-data") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:34.931131 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.931109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:34.931210 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.931135    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h" (OuterVolumeSpecName: "kube-api-access-79v5h") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "kube-api-access-79v5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:34.931210 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.931197    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token" (OuterVolumeSpecName: "istio-token") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:34.931318 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:34.931303    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "46867301-08d7-4fd3-ab7b-87672734e344" (UID: "46867301-08d7-4fd3-ab7b-87672734e344"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:21:35.030337 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030294    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79v5h\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-kube-api-access-79v5h\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.030337 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030330    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46867301-08d7-4fd3-ab7b-87672734e344-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.030337 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030341    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46867301-08d7-4fd3-ab7b-87672734e344-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.030337 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030349    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.030604 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030358    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.030604 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.030366    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46867301-08d7-4fd3-ab7b-87672734e344-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:35.134540 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.134503    2565 generic.go:356] "Generic (PLEG): container finished" podID="46867301-08d7-4fd3-ab7b-87672734e344" containerID="4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8" exitCode=0
Jul 29 22:21:35.134679 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.134548    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"
Jul 29 22:21:35.134679 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.134560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerDied","Data":"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"}
Jul 29 22:21:35.134679 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.134610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4" event={"ID":"46867301-08d7-4fd3-ab7b-87672734e344","Type":"ContainerDied","Data":"542a8d158e49196824079bb25a4298ac53158b63ee589f5446d87e124f22fda6"}
Jul 29 22:21:35.134679 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.134632    2565 scope.go:117] "RemoveContainer" containerID="4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"
Jul 29 22:21:35.145200 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.145184    2565 scope.go:117] "RemoveContainer" containerID="2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"
Jul 29 22:21:35.154431 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.154397    2565 scope.go:117] "RemoveContainer" containerID="4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"
Jul 29 22:21:35.154709 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:35.154685    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8\": container with ID starting with 4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8 not found: ID does not exist" containerID="4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"
Jul 29 22:21:35.154758 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.154720    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8"} err="failed to get container status \"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8\": rpc error: code = NotFound desc = could not find container \"4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8\": container with ID starting with 4a4069da2e7f72ab530d363826966a157a0161e1d20b01eccc90098dbcd21fe8 not found: ID does not exist"
Jul 29 22:21:35.154758 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.154744    2565 scope.go:117] "RemoveContainer" containerID="2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"
Jul 29 22:21:35.154959 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:35.154935    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b\": container with ID starting with 2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b not found: ID does not exist" containerID="2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"
Jul 29 22:21:35.155006 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.154969    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b"} err="failed to get container status \"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b\": rpc error: code = NotFound desc = could not find container \"2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b\": container with ID starting with 2a730db6d846a41031dcb86b88bddc99e09b920a7319eebc7bc05a1eb502724b not found: ID does not exist"
Jul 29 22:21:35.159577 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.159542    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:21:35.162901 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:35.162879    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-b26bbb-openshift-default-fbd47bcd4-pn2l4"]
Jul 29 22:21:36.055631 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:36.055593    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46867301-08d7-4fd3-ab7b-87672734e344" path="/var/lib/kubelet/pods/46867301-08d7-4fd3-ab7b-87672734e344/volumes"
Jul 29 22:21:40.683992 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:40.683960    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:21:40.684379 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:40.684199    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" podUID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" containerName="maas-api" containerID="cri-o://51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281" gracePeriod=30
Jul 29 22:21:40.955446 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:40.955420    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:21:41.095997 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.095954    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2x88\" (UniqueName: \"kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88\") pod \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") "
Jul 29 22:21:41.096200 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.096132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") pod \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") "
Jul 29 22:21:41.096200 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.096160    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca\") pod \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\" (UID: \"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0\") "
Jul 29 22:21:41.096598 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.096572    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" (UID: "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:21:41.098166 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.098133    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" (UID: "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:21:41.098273 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.098208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88" (OuterVolumeSpecName: "kube-api-access-c2x88") pod "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" (UID: "63a8a3cb-d1b8-4575-b42a-8e33717ff6a0"). InnerVolumeSpecName "kube-api-access-c2x88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:41.167573 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.167539    2565 generic.go:356] "Generic (PLEG): container finished" podID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" containerID="51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281" exitCode=0
Jul 29 22:21:41.167748 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.167587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"
Jul 29 22:21:41.167748 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.167611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" event={"ID":"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0","Type":"ContainerDied","Data":"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"}
Jul 29 22:21:41.167748 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.167657    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp" event={"ID":"63a8a3cb-d1b8-4575-b42a-8e33717ff6a0","Type":"ContainerDied","Data":"e24a2b0c0bb92415f53bb1b7c4148556ecc5fd29ebae10e72e934706b61af29d"}
Jul 29 22:21:41.167748 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.167678    2565 scope.go:117] "RemoveContainer" containerID="51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"
Jul 29 22:21:41.176745 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.176728    2565 scope.go:117] "RemoveContainer" containerID="51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"
Jul 29 22:21:41.177037 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:41.177015    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281\": container with ID starting with 51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281 not found: ID does not exist" containerID="51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"
Jul 29 22:21:41.177134 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.177060    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281"} err="failed to get container status \"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281\": rpc error: code = NotFound desc = could not find container \"51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281\": container with ID starting with 51416baa263a2d9a0b287b3d6cac2328252ea7f484be3622f041240ef6091281 not found: ID does not exist"
Jul 29 22:21:41.192482 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.192401    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:21:41.196587 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.196562    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-c9eccd-6cffd4d568-whmvp"]
Jul 29 22:21:41.197028 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.197010    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:41.197093 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.197032    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:41.197093 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:41.197054    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2x88\" (UniqueName: \"kubernetes.io/projected/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0-kube-api-access-c2x88\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:42.055841 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:42.055805    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" path="/var/lib/kubelet/pods/63a8a3cb-d1b8-4575-b42a-8e33717ff6a0/volumes"
Jul 29 22:21:47.696614 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:47.696579    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:21:47.697130 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:47.696879    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy" containerID="cri-o://d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161" gracePeriod=30
Jul 29 22:21:52.956949 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:52.956924    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:21:53.122190 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122194    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgnw5\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122302    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122372 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122354    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122434    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:53.122630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122561    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:53.122840 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122635    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo\") pod \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\" (UID: \"e04582e1-a195-4899-b1d3-6ef1f7538bc9\") "
Jul 29 22:21:53.122840 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122677    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:53.122943 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.122836    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:21:53.123139 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.123106    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.123139 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.123137    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.123341 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.123152    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.123341 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.123166    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.123341 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.123241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data" (OuterVolumeSpecName: "istio-data") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:53.124652 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.124623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token" (OuterVolumeSpecName: "istio-token") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:53.124767 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.124662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:21:53.124943 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.124921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:21:53.124943 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.124935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5" (OuterVolumeSpecName: "kube-api-access-kgnw5") pod "e04582e1-a195-4899-b1d3-6ef1f7538bc9" (UID: "e04582e1-a195-4899-b1d3-6ef1f7538bc9"). InnerVolumeSpecName "kube-api-access-kgnw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:53.224205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.224170    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgnw5\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-kube-api-access-kgnw5\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.224205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.224201    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.224205 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.224211    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.224438 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.224220    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.224438 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.224227    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e04582e1-a195-4899-b1d3-6ef1f7538bc9-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:53.236524 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.236489    2565 generic.go:356] "Generic (PLEG): container finished" podID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerID="d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161" exitCode=0
Jul 29 22:21:53.236673 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.236541    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"
Jul 29 22:21:53.236673 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.236574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerDied","Data":"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"}
Jul 29 22:21:53.236673 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.236630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69" event={"ID":"e04582e1-a195-4899-b1d3-6ef1f7538bc9","Type":"ContainerDied","Data":"e4576e58c73ac7cd02e20736938d9453bb8ffe73de20151311b7497237d182be"}
Jul 29 22:21:53.237227 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.236660    2565 scope.go:117] "RemoveContainer" containerID="d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"
Jul 29 22:21:53.247542 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.247507    2565 scope.go:117] "RemoveContainer" containerID="59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"
Jul 29 22:21:53.256194 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.256172    2565 scope.go:117] "RemoveContainer" containerID="d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"
Jul 29 22:21:53.256460 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:53.256437    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161\": container with ID starting with d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161 not found: ID does not exist" containerID="d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"
Jul 29 22:21:53.256591 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.256469    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161"} err="failed to get container status \"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161\": rpc error: code = NotFound desc = could not find container \"d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161\": container with ID starting with d36d204e2708472dd67e44021c43f80c945ed0544396c391be968a013e084161 not found: ID does not exist"
Jul 29 22:21:53.256591 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.256488    2565 scope.go:117] "RemoveContainer" containerID="59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"
Jul 29 22:21:53.256781 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:53.256746    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8\": container with ID starting with 59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8 not found: ID does not exist" containerID="59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"
Jul 29 22:21:53.256843 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.256786    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8"} err="failed to get container status \"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8\": rpc error: code = NotFound desc = could not find container \"59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8\": container with ID starting with 59829227f40400a2ddcebf68da7c2acfbd2e8038098857ebb091150af003cdf8 not found: ID does not exist"
Jul 29 22:21:53.259880 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.259857    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:21:53.266195 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:53.266174    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-c9eccd-openshift-default-8789b56b4-k4p69"]
Jul 29 22:21:54.055388 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:54.055351    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" path="/var/lib/kubelet/pods/e04582e1-a195-4899-b1d3-6ef1f7538bc9/volumes"
Jul 29 22:21:58.594511 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.594475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:21:58.594977 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.594746    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" podUID="7dab7830-950a-439a-a7b1-cea70c9b29fc" containerName="maas-api" containerID="cri-o://f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0" gracePeriod=30
Jul 29 22:21:58.845239 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.845170    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:21:58.878281 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.878242    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca\") pod \"7dab7830-950a-439a-a7b1-cea70c9b29fc\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") "
Jul 29 22:21:58.878480 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.878307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bsw6\" (UniqueName: \"kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6\") pod \"7dab7830-950a-439a-a7b1-cea70c9b29fc\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") "
Jul 29 22:21:58.878480 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.878361    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") pod \"7dab7830-950a-439a-a7b1-cea70c9b29fc\" (UID: \"7dab7830-950a-439a-a7b1-cea70c9b29fc\") "
Jul 29 22:21:58.878895 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.878780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7dab7830-950a-439a-a7b1-cea70c9b29fc" (UID: "7dab7830-950a-439a-a7b1-cea70c9b29fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:21:58.881586 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.881553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7dab7830-950a-439a-a7b1-cea70c9b29fc" (UID: "7dab7830-950a-439a-a7b1-cea70c9b29fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 29 22:21:58.881730 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.881681    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6" (OuterVolumeSpecName: "kube-api-access-2bsw6") pod "7dab7830-950a-439a-a7b1-cea70c9b29fc" (UID: "7dab7830-950a-439a-a7b1-cea70c9b29fc"). InnerVolumeSpecName "kube-api-access-2bsw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:21:58.979442 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.979406    2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dab7830-950a-439a-a7b1-cea70c9b29fc-openshift-service-ca\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:58.979442 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.979438    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bsw6\" (UniqueName: \"kubernetes.io/projected/7dab7830-950a-439a-a7b1-cea70c9b29fc-kube-api-access-2bsw6\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:58.979442 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:58.979450    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7dab7830-950a-439a-a7b1-cea70c9b29fc-maas-api-tls\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:21:59.271425 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.271394    2565 generic.go:356] "Generic (PLEG): container finished" podID="7dab7830-950a-439a-a7b1-cea70c9b29fc" containerID="f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0" exitCode=0
Jul 29 22:21:59.271597 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.271438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"
Jul 29 22:21:59.271597 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.271469    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" event={"ID":"7dab7830-950a-439a-a7b1-cea70c9b29fc","Type":"ContainerDied","Data":"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"}
Jul 29 22:21:59.271597 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.271512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22" event={"ID":"7dab7830-950a-439a-a7b1-cea70c9b29fc","Type":"ContainerDied","Data":"e18a9538a0f3d1d7672db7aa262c90da521954493140a455ac09541751fd7e57"}
Jul 29 22:21:59.271597 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.271533    2565 scope.go:117] "RemoveContainer" containerID="f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"
Jul 29 22:21:59.280672 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.280655    2565 scope.go:117] "RemoveContainer" containerID="f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"
Jul 29 22:21:59.280925 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:21:59.280904    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0\": container with ID starting with f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0 not found: ID does not exist" containerID="f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"
Jul 29 22:21:59.280992 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.280932    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0"} err="failed to get container status \"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0\": rpc error: code = NotFound desc = could not find container \"f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0\": container with ID starting with f8885f6f46a03cd540b9b9bae03baeaa0fb0a9d1260c91cf35542a00fdbfc3e0 not found: ID does not exist"
Jul 29 22:21:59.294593 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.294568    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:21:59.299554 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:21:59.299535    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ca895e-74845c5757-bgs22"]
Jul 29 22:22:00.055344 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:00.055308    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dab7830-950a-439a-a7b1-cea70c9b29fc" path="/var/lib/kubelet/pods/7dab7830-950a-439a-a7b1-cea70c9b29fc/volumes"
Jul 29 22:22:05.512641 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:05.512608    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:22:05.513133 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:05.512840    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy" containerID="cri-o://ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2" gracePeriod=30
Jul 29 22:22:10.768469 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.768435    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:22:10.903307 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903307 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903258    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903307 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903307 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903353    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903428    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psdx7\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903548    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data\") pod \"deb49e65-594a-402f-a4df-4ab35a491048\" (UID: \"deb49e65-594a-402f-a4df-4ab35a491048\") "
Jul 29 22:22:10.903594 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903542    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:22:10.903886 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903646    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:22:10.903886 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:22:10.904003 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903949    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-certs\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:10.904003 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903967    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-credential-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:10.904003 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.903978    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-workload-socket\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:10.904627 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.904598    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data" (OuterVolumeSpecName: "istio-data") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:22:10.904747 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.904663    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 29 22:22:10.905850 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.905825    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token" (OuterVolumeSpecName: "istio-token") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:22:10.905951 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.905853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 29 22:22:10.905951 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.905858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 29 22:22:10.906034 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:10.906008    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7" (OuterVolumeSpecName: "kube-api-access-psdx7") pod "deb49e65-594a-402f-a4df-4ab35a491048" (UID: "deb49e65-594a-402f-a4df-4ab35a491048"). InnerVolumeSpecName "kube-api-access-psdx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 29 22:22:11.004727 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004692    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-istio-token\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.004727 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004724    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb49e65-594a-402f-a4df-4ab35a491048-istiod-ca-cert\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.004727 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004733    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-envoy\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.004727 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004741    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb49e65-594a-402f-a4df-4ab35a491048-istio-podinfo\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.004994 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004753    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdx7\" (UniqueName: \"kubernetes.io/projected/deb49e65-594a-402f-a4df-4ab35a491048-kube-api-access-psdx7\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.004994 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.004767    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb49e65-594a-402f-a4df-4ab35a491048-istio-data\") on node \"ip-10-0-129-123.ec2.internal\" DevicePath \"\""
Jul 29 22:22:11.335847 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.335809    2565 generic.go:356] "Generic (PLEG): container finished" podID="deb49e65-594a-402f-a4df-4ab35a491048" containerID="ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2" exitCode=0
Jul 29 22:22:11.336036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.335872    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"
Jul 29 22:22:11.336036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.335893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerDied","Data":"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"}
Jul 29 22:22:11.336036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.335955    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv" event={"ID":"deb49e65-594a-402f-a4df-4ab35a491048","Type":"ContainerDied","Data":"2dedbea15c2fbd52ea3ed8fa837935c642ceb29021194c7803560c30a011c9af"}
Jul 29 22:22:11.336036 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.335972    2565 scope.go:117] "RemoveContainer" containerID="ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"
Jul 29 22:22:11.345142 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.345122    2565 scope.go:117] "RemoveContainer" containerID="ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"
Jul 29 22:22:11.353589 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.353564    2565 scope.go:117] "RemoveContainer" containerID="ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"
Jul 29 22:22:11.353833 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:22:11.353817    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2\": container with ID starting with ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2 not found: ID does not exist" containerID="ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"
Jul 29 22:22:11.353886 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.353841    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2"} err="failed to get container status \"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2\": rpc error: code = NotFound desc = could not find container \"ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2\": container with ID starting with ce0a673560850119cb7fac98da4f7aa20927ff364bbd7f61521e8a5b632c6bf2 not found: ID does not exist"
Jul 29 22:22:11.353886 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.353857    2565 scope.go:117] "RemoveContainer" containerID="ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"
Jul 29 22:22:11.354109 ip-10-0-129-123 kubenswrapper[2565]: E0729 22:22:11.354089    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c\": container with ID starting with ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c not found: ID does not exist" containerID="ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"
Jul 29 22:22:11.354181 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.354118    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c"} err="failed to get container status \"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c\": rpc error: code = NotFound desc = could not find container \"ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c\": container with ID starting with ee224f3453f823faa793da923a6e6984e96bfbb742a0272630429b2a1745449c not found: ID does not exist"
Jul 29 22:22:11.357703 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.357679    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:22:11.361478 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:11.361455    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-ca895e-openshift-default-585876c78d-wqnnv"]
Jul 29 22:22:12.055169 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:12.055136    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb49e65-594a-402f-a4df-4ab35a491048" path="/var/lib/kubelet/pods/deb49e65-594a-402f-a4df-4ab35a491048/volumes"
Jul 29 22:22:12.875217 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:12.875175    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z6hq8_69d45d59-fdb0-44a7-98a6-a41677f79a0b/manager/0.log"
Jul 29 22:22:12.999277 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:12.999235    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-njl7k_d6be8fa7-28a8-45c4-8847-d94ee8e48c81/manager/0.log"
Jul 29 22:22:13.128185 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:13.128094    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-656b7fbfd4-v4htm_9b7602e9-e020-45e5-b61c-e733d74d6067/manager/0.log"
Jul 29 22:22:13.764163 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:13.764134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-pwbq8_107ebb15-70e2-4433-8ac8-c388bfa2f010/copy-manifests/0.log"
Jul 29 22:22:13.770550 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:13.770522    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-pwbq8_107ebb15-70e2-4433-8ac8-c388bfa2f010/manager/0.log"
Jul 29 22:22:14.141596 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:14.141510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-79659c9889-xnwzp_6d6302d9-03b4-4d77-a431-7723f93e5ded/maas-api/0.log"
Jul 29 22:22:14.262133 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:14.262103    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756025-nqhwh_8fc4f2e7-dfa1-452d-93b0-4c3b39cba2c7/cleanup/0.log"
Jul 29 22:22:14.381118 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:14.381089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29756040-rc6bt_e626741b-e008-4b88-96a1-d888543f418a/cleanup/0.log"
Jul 29 22:22:14.612266 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:14.612238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rtk9h_a597dacb-843c-4c6b-a375-9c0c89a43944/postgres/0.log"
Jul 29 22:22:17.152944 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:17.152861    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n_c686c0c7-6a0c-4ecd-93ae-c9db3a21b241/istio-proxy/0.log"
Jul 29 22:22:17.624952 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:17.624915    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/1.log"
Jul 29 22:22:17.746103 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:17.746073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pt8x4_44d7037b-7db0-4b22-ac71-33a4fc9d2973/payload-pre-processing/0.log"
Jul 29 22:22:17.865801 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:17.865759    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:22:18.347007 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:18.346980    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-x7jf8_ad4a5a4d-45d7-45c3-8ec8-f211a0a8b27a/main/0.log"
Jul 29 22:22:18.474227 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:18.474185    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-j4tnd_7c1a61af-d182-44ad-afd4-ac75e1e04e50/main/0.log"
Jul 29 22:22:18.827270 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:18.827240    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6dph8_0735801d-f5dc-411f-b5db-7aad83eb8922/main/0.log"
Jul 29 22:22:27.810221 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:27.810186    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-57rmr_f78c6650-9271-48e4-b274-cfd7e4d542b3/global-pull-secret-syncer/0.log"
Jul 29 22:22:28.072642 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:28.072563    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z9scg_d54df4e4-1468-4fdf-91cf-2ad540cacf25/konnectivity-agent/0.log"
Jul 29 22:22:28.097563 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:28.097529    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-123.ec2.internal_fc048883d1237889be55fccbd385befa/haproxy/0.log"
Jul 29 22:22:35.332658 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.332634    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/alertmanager/0.log"
Jul 29 22:22:35.356682 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.356650    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/config-reloader/0.log"
Jul 29 22:22:35.378677 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.378652    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/kube-rbac-proxy-web/0.log"
Jul 29 22:22:35.401545 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.401516    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/kube-rbac-proxy/0.log"
Jul 29 22:22:35.424187 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.424154    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/kube-rbac-proxy-metric/0.log"
Jul 29 22:22:35.445971 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.445938    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/prom-label-proxy/0.log"
Jul 29 22:22:35.467610 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.467585    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f3378642-6c50-455f-9d45-07c01a88bb20/init-config-reloader/0.log"
Jul 29 22:22:35.553813 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.553769    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5q5w4_84b77d03-d1fe-423d-91c1-3861873b302a/kube-state-metrics/0.log"
Jul 29 22:22:35.572833 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.572795    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5q5w4_84b77d03-d1fe-423d-91c1-3861873b302a/kube-rbac-proxy-main/0.log"
Jul 29 22:22:35.595903 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.595826    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-5q5w4_84b77d03-d1fe-423d-91c1-3861873b302a/kube-rbac-proxy-self/0.log"
Jul 29 22:22:35.673140 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.673117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-7dzzt_aa80b862-1c98-4243-85dd-26ba90602d0d/monitoring-plugin/0.log"
Jul 29 22:22:35.803130 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.803097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-84wmf_27f3633c-8a0c-421f-9693-b3ee90c448b9/node-exporter/0.log"
Jul 29 22:22:35.827134 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.827104    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-84wmf_27f3633c-8a0c-421f-9693-b3ee90c448b9/kube-rbac-proxy/0.log"
Jul 29 22:22:35.850311 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:35.850226    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-84wmf_27f3633c-8a0c-421f-9693-b3ee90c448b9/init-textfile/0.log"
Jul 29 22:22:36.057324 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.057298    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/prometheus/0.log"
Jul 29 22:22:36.085621 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.085597    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/config-reloader/0.log"
Jul 29 22:22:36.148917 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.148842    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/thanos-sidecar/0.log"
Jul 29 22:22:36.185833 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.185815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/kube-rbac-proxy-web/0.log"
Jul 29 22:22:36.212219 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.212192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/kube-rbac-proxy/0.log"
Jul 29 22:22:36.245239 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.245215    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/kube-rbac-proxy-thanos/0.log"
Jul 29 22:22:36.270415 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.270389    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d9166be1-bbde-4752-b13e-0dd7cf04a5cb/init-config-reloader/0.log"
Jul 29 22:22:36.393152 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.393121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784fff9fbd-9bwcq_f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8/telemeter-client/0.log"
Jul 29 22:22:36.420946 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.420917    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784fff9fbd-9bwcq_f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8/reload/0.log"
Jul 29 22:22:36.444569 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:36.444543    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-784fff9fbd-9bwcq_f88a5fe2-e7f9-46b2-94c4-d8f6accec4c8/kube-rbac-proxy/0.log"
Jul 29 22:22:38.330557 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330522    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"]
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330945    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a127c57c-1e85-4399-bb01-a5125f07b9d9" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330951    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a127c57c-1e85-4399-bb01-a5125f07b9d9" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330959    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dab7830-950a-439a-a7b1-cea70c9b29fc" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330964    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dab7830-950a-439a-a7b1-cea70c9b29fc" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330970    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330977    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" containerName="maas-api"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330988    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.330996    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331014    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331014 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331022    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331032    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331055    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331067    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331085    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331092    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331104    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331112    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331120    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331125    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331226    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63a8a3cb-d1b8-4575-b42a-8e33717ff6a0" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331241    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331246    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331252    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a127c57c-1e85-4399-bb01-a5125f07b9d9" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331259    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dab7830-950a-439a-a7b1-cea70c9b29fc" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331268    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9804e4cd-8245-4331-94e2-c6e48611c429" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331275    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb49e65-594a-402f-a4df-4ab35a491048" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331283    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e698bfb-2567-4f6b-ac61-4f5483335f8f" containerName="maas-api"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331293    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331303    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e04582e1-a195-4899-b1d3-6ef1f7538bc9" containerName="istio-proxy"
Jul 29 22:22:38.331634 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.331310    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="46867301-08d7-4fd3-ab7b-87672734e344" containerName="istio-proxy"
Jul 29 22:22:38.336149 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.336130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.338893 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.338874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqkkn\"/\"kube-root-ca.crt\""
Jul 29 22:22:38.339760 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.339730    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jqkkn\"/\"default-dockercfg-vhgbw\""
Jul 29 22:22:38.339910 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.339799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqkkn\"/\"openshift-service-ca.crt\""
Jul 29 22:22:38.345255 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.345226    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"]
Jul 29 22:22:38.373992 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.373957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzx5t\" (UniqueName: \"kubernetes.io/projected/d4551321-3638-4943-84f4-4902edae5855-kube-api-access-pzx5t\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.374200 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.374001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-sys\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.374200 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.374160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-podres\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.374327 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.374264    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-lib-modules\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.374327 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.374302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-proc\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475590 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-podres\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-lib-modules\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-proc\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475682    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzx5t\" (UniqueName: \"kubernetes.io/projected/d4551321-3638-4943-84f4-4902edae5855-kube-api-access-pzx5t\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-sys\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475740    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-podres\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.475773 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475766    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-proc\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.476006 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475821    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-lib-modules\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.476006 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.475827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4551321-3638-4943-84f4-4902edae5855-sys\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.484504 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.484470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzx5t\" (UniqueName: \"kubernetes.io/projected/d4551321-3638-4943-84f4-4902edae5855-kube-api-access-pzx5t\") pod \"perf-node-gather-daemonset-wss6x\" (UID: \"d4551321-3638-4943-84f4-4902edae5855\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.647859 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.647768    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:38.809753 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.809721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"]
Jul 29 22:22:38.811313 ip-10-0-129-123 kubenswrapper[2565]: W0729 22:22:38.811286    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd4551321_3638_4943_84f4_4902edae5855.slice/crio-dbb6ca307ebf768adac88567eadaeb2db3181e18dfab522acbb1efeaa334b07f WatchSource:0}: Error finding container dbb6ca307ebf768adac88567eadaeb2db3181e18dfab522acbb1efeaa334b07f: Status 404 returned error can't find the container with id dbb6ca307ebf768adac88567eadaeb2db3181e18dfab522acbb1efeaa334b07f
Jul 29 22:22:38.954889 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.954857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6945bf6885-kgn59_06c71a14-8599-4975-9dae-55dbf5feb873/console/0.log"
Jul 29 22:22:38.992562 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:38.992529    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-6wc8p_d83b763d-5d3c-42c7-bd05-b7b3992a1e81/download-server/0.log"
Jul 29 22:22:39.487185 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:39.487143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x" event={"ID":"d4551321-3638-4943-84f4-4902edae5855","Type":"ContainerStarted","Data":"61c06c9ad2d2e1d3d73542f13934a59ef6d908df20b2d109ee2f635a4078a559"}
Jul 29 22:22:39.487672 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:39.487198    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x" event={"ID":"d4551321-3638-4943-84f4-4902edae5855","Type":"ContainerStarted","Data":"dbb6ca307ebf768adac88567eadaeb2db3181e18dfab522acbb1efeaa334b07f"}
Jul 29 22:22:39.487672 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:39.487279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:39.507443 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:39.507389    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x" podStartSLOduration=1.5073739800000001 podStartE2EDuration="1.50737398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:22:39.505103054 +0000 UTC m=+3975.976682126" watchObservedRunningTime="2026-07-29 22:22:39.50737398 +0000 UTC m=+3975.978953078"
Jul 29 22:22:39.544274 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:39.544251    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8g6g6_d5ea33ae-e191-4d92-ba22-651b3729928f/volume-data-source-validator/0.log"
Jul 29 22:22:40.637678 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:40.637651    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g2nvr_05830715-42d5-461f-942c-f14f4290a2a1/dns/0.log"
Jul 29 22:22:40.656763 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:40.656735    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g2nvr_05830715-42d5-461f-942c-f14f4290a2a1/kube-rbac-proxy/0.log"
Jul 29 22:22:40.777435 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:40.777408    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xgrr6_8f9ce67f-8d99-4397-8fe6-395e53f23440/dns-node-resolver/0.log"
Jul 29 22:22:41.320367 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:41.320332    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ljfcj_c1a702f0-7382-4c43-b759-1466980f155e/node-ca/0.log"
Jul 29 22:22:42.338035 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:42.338004    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n_c686c0c7-6a0c-4ecd-93ae-c9db3a21b241/istio-proxy/0.log"
Jul 29 22:22:42.575178 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:42.575145    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/0.log"
Jul 29 22:22:42.600370 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:42.600278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/1.log"
Jul 29 22:22:42.629937 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:42.629900    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pt8x4_44d7037b-7db0-4b22-ac71-33a4fc9d2973/payload-pre-processing/0.log"
Jul 29 22:22:42.660904 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:42.660868    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:22:43.526936 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:43.526908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mw6qh_87058e58-2b75-43ed-9991-361cb06de8fe/serve-healthcheck-canary/0.log"
Jul 29 22:22:44.110872 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:44.110845    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvf9r_aba3f655-06ad-4d05-8676-230b17e4c922/kube-rbac-proxy/0.log"
Jul 29 22:22:44.132396 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:44.132367    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvf9r_aba3f655-06ad-4d05-8676-230b17e4c922/exporter/0.log"
Jul 29 22:22:44.153865 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:44.153841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cvf9r_aba3f655-06ad-4d05-8676-230b17e4c922/extractor/0.log"
Jul 29 22:22:45.501662 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:45.501631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wss6x"
Jul 29 22:22:46.362442 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:46.362408    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-z6hq8_69d45d59-fdb0-44a7-98a6-a41677f79a0b/manager/0.log"
Jul 29 22:22:46.581023 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:46.580989    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-njl7k_d6be8fa7-28a8-45c4-8847-d94ee8e48c81/manager/0.log"
Jul 29 22:22:46.731575 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:46.731543    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-656b7fbfd4-v4htm_9b7602e9-e020-45e5-b61c-e733d74d6067/manager/0.log"
Jul 29 22:22:47.079107 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:47.079021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-pwbq8_107ebb15-70e2-4433-8ac8-c388bfa2f010/manager/0.log"
Jul 29 22:22:47.100513 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:47.100485    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-pwbq8_107ebb15-70e2-4433-8ac8-c388bfa2f010/copy-manifests/0.log"
Jul 29 22:22:48.520208 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.520121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5999ff5f6f-bg8lv_c1f8c9a5-132d-499e-9301-3db3628bcf2a/manager/0.log"
Jul 29 22:22:48.678840 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.678807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n_c686c0c7-6a0c-4ecd-93ae-c9db3a21b241/istio-proxy/0.log"
Jul 29 22:22:48.888876 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.888779    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/0.log"
Jul 29 22:22:48.915240 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.915212    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/1.log"
Jul 29 22:22:48.936314 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.936285    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pt8x4_44d7037b-7db0-4b22-ac71-33a4fc9d2973/payload-pre-processing/0.log"
Jul 29 22:22:48.956630 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:48.956604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:22:50.342464 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:50.342431    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-pw27n_c686c0c7-6a0c-4ecd-93ae-c9db3a21b241/istio-proxy/0.log"
Jul 29 22:22:50.540438 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:50.540394    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/0.log"
Jul 29 22:22:50.568580 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:50.568542    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-mjv7r_3bc54791-82ee-46cc-bc56-3497049bfef7/istio-proxy/1.log"
Jul 29 22:22:50.590325 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:50.590297    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-pt8x4_44d7037b-7db0-4b22-ac71-33a4fc9d2973/payload-pre-processing/0.log"
Jul 29 22:22:50.611193 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:50.611167    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-2kvc7_d0070d94-0d67-4f4d-9292-78b305332e4e/payload-processing/0.log"
Jul 29 22:22:53.673234 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:53.673201    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wcx7v_7d1b8018-59b9-4a8e-a718-4b7feec19c36/migrator/0.log"
Jul 29 22:22:53.694321 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:53.694294    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-wcx7v_7d1b8018-59b9-4a8e-a718-4b7feec19c36/graceful-termination/0.log"
Jul 29 22:22:55.352286 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.352250    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/kube-multus-additional-cni-plugins/0.log"
Jul 29 22:22:55.375015 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.374983    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/egress-router-binary-copy/0.log"
Jul 29 22:22:55.400841 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.400815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/cni-plugins/0.log"
Jul 29 22:22:55.423204 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.423177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/bond-cni-plugin/0.log"
Jul 29 22:22:55.449523 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.449495    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/routeoverride-cni/0.log"
Jul 29 22:22:55.477002 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.476972    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/whereabouts-cni-bincopy/0.log"
Jul 29 22:22:55.504952 ip-10-0-129-123 kubenswrapper[2565]: I0729 22:22:55.504927    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fz857_a6243582-85a5-456a-8acb-a76632024c98/whereabouts-cni/0.log"